PCAP Files Filling up HardDrive

[DJNAT10]

Good Morning,

I submitted an issue approximately three weeks ago—apologies for the delayed follow-up. Despite implementing the recommended steps including adding debug lines to the configuration files, our PCAP files are still not being deleted. Upon reviewing the Arkime logs, we are encountering the following error:

arkime-1 | EXPIRE - check device node: HOUMALCOMP02 free: 6956.841846466064 freeSpaceG: 7618.639091491699
arkime-1 | EXPIRE - check device node: arkime free: 6956.841846466064 freeSpaceG: 7618.639091491699
arkime-1 | EXPIRE - check device node: HOUMALCOMP02-upload free: 6956.841846466064 freeSpaceG: 7618.639091491699
arkime-1 | EXPIRE - device [ 'HOUMALCOMP02', 'arkime', 'HOUMALCOMP02-upload' ] { '/data/pcap/processed': {} } 7618.639091491699
arkime-1 | EXPIRE - device query {
arkime-1 | "_source": [
arkime-1 | "num",
arkime-1 | "name",
arkime-1 | "first",
arkime-1 | "size",
arkime-1 | "node",
arkime-1 | "indexFilename"
arkime-1 | ],
arkime-1 | "from": "10",
arkime-1 | "size": 500,
arkime-1 | "query": {
arkime-1 | "bool": {
arkime-1 | "filter": [
arkime-1 | {
arkime-1 | "terms": {
arkime-1 | "node": [
arkime-1 | "HOUMALCOMP02",
arkime-1 | "arkime",
arkime-1 | "HOUMALCOMP02-upload"
arkime-1 | ]
arkime-1 | }
arkime-1 | },
arkime-1 | {
arkime-1 | "bool": {
arkime-1 | "should": [
arkime-1 | {
arkime-1 | "wildcard": {
arkime-1 | "name": "/data/pcap/processed/*"
arkime-1 | }
arkime-1 | }
arkime-1 | ]
arkime-1 | }
arkime-1 | }
arkime-1 | ],
arkime-1 | "must_not": {
arkime-1 | "term": {
arkime-1 | "locked": 1
arkime-1 | }
arkime-1 | }
arkime-1 | }
arkime-1 | },
arkime-1 | "sort": {
arkime-1 | "first": {
arkime-1 | "order": "asc"
arkime-1 | }
arkime-1 | }
arkime-1 | }
arkime-1 | EXPIRE - device results null {
arkime-1 | "took": 7,
arkime-1 | "timed_out": false,
arkime-1 | "_shards": {
arkime-1 | "total": 2,
arkime-1 | "successful": 2,
arkime-1 | "skipped": 0,
arkime-1 | "failed": 0
arkime-1 | },
arkime-1 | "hits": {
arkime-1 | "total": 10,
arkime-1 | "max_score": null,
arkime-1 | "hits": []
arkime-1 | }
arkime-1 | }
arkime-1 | EXPIRE WARNING - not deleting any files since 10 <= 10 minimum files per node. Your disk(s) may fill!!! See https://arkime.com/faq#pcap-deletion

We would love to hear your input.

Thank you,
Natasha Jones

[mmguero]

• What do you have set for the following:
  • config/arkime.env
    • ARKIME_FREESPACEG
    • MANAGE_PCAP_FILES?
  • config/pcap-capture.env
    • PCAP_ROTATE_MEGABYTES
• How big is the disk, and how much free space is there to begin with prior to PCAPs being written?

As the error message says, and the link from the error message: "The viewer process will always leave at least 10 PCAP files on the disk, so make sure there is room for at least maxFileSizeG * 10 capture files on disk"

See this documentation as well for some background.

[DJNAT10]

For config/arkime.env:

• ARKIME_FREESPACEG=60%
• MANAGE_PCAP_FILES=true

For config/pcap-capture.env:

• PCAP_ROTATE_MEGABYTES=500

Our server is currently down, so I'm waiting for it to reboot before I can retrieve the disk space configuration variables.

[mmguero]

60% means "delete PCAP files if the amount of free space on the file system falls below 60%".

With PCAP_ROTATE_MEGABYTES=500, when i do this conversion for populating arkime's config.ini Arkime will end up with a maxFileSizeG=1 value, which means, according to that error message "The viewer process will always leave at least 10 PCAP files on the disk, so make sure there is room for at least maxFileSizeG * 10 capture files on disk," so there needs to be room for at least 10 1-gigbayte files on the disk before anything gets deleted.

Are the PCAP files that are in question live-captured PCAP files, or uploaded via the upload interface?