Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
22 out of 60 is failed (threshold=3) [................XX..XXXXXX.XXXXXX.XXXXXXXX..................] duration:
433.680482ms trace:
https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8
FAIL (2025-12-11 00:03:59 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload
Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload
Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments...
Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false
Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment...
Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment.
Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment
Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments...
Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments...
Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 433.706028ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [....X....X...X........X...X.................................] duration:
523.871225ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-11 00:03:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 523.898878ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.XXX.X......................................................] duration:
422.821199ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-11 00:03:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 423.068901ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [........XXX.................................................] duration:
559.588995ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-11 00:03:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 559.61891ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [............XXXXXXX.XXXXXXXXXXX.XXXXXXX.....................] duration:
458.256927ms trace:
https://jaeger.dev.storj.io/trace/4d0434aacaaec70b
FAIL (2025-12-11 00:03:26 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
.578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment
Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments...
Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments...
Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 458.285204ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [.............XXXXXXX.XXXXXXXXXXX.XXXXXXX....................] duration:
570.720996ms trace:
https://jaeger.dev.storj.io/trace/4d0434aacaaec70b
FAIL (2025-12-11 00:03:02 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
.578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment
Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments...
Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments...
Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 570.827053ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
22 out of 60 is failed (threshold=3) [.................XX..XXXXXX.XXXXXX.XXXXXXXX.................] duration:
399.101534ms trace:
https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8
FAIL (2025-12-11 00:02:57 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload
Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload
Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments...
Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false
Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment...
Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment.
Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment
Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments...
Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments...
Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 399.201839ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....X....X...X........X...X................................] duration:
481.195073ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-11 00:02:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 481.724771ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..XXX.X.....................................................] duration:
588.804682ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-11 00:02:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 588.829014ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.........XXX................................................] duration:
528.456157ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-11 00:02:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 528.502008ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [..............XXXXXXX.XXXXXXXXXXX.XXXXXXX...................] duration:
464.604171ms trace:
https://jaeger.dev.storj.io/trace/4d0434aacaaec70b
FAIL (2025-12-11 00:02:21 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
.578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment
Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments...
Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments...
Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 464.63221ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...XXX.X....................................................] duration:
386.591791ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-11 00:02:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 386.685904ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [......X....X...X........X...X...............................] duration:
479.875661ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-11 00:01:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 479.904998ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [...............XXXXXXX.XXXXXXXXXXX.XXXXXXX..................] duration:
488.976289ms trace:
https://jaeger.dev.storj.io/trace/4d0434aacaaec70b
FAIL (2025-12-11 00:01:52 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
.578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment
Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments...
Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments...
Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 489.005003ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [....XXX.X...................................................] duration:
606.235557ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-11 00:01:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 606.262935ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
22 out of 60 is failed (threshold=3) [..................XX..XXXXXX.XXXXXX.XXXXXXXX................] duration:
407.58851ms trace:
https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8
FAIL (2025-12-11 00:01:43 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload
Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload
Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments...
Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false
Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment...
Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment.
Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment
Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments...
Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments...
Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 407.615459ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.....XXX.X..................................................] duration:
412.620769ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-11 00:01:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 413.375809ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..........XXX...............................................] duration:
552.379097ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-11 00:01:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 552.488862ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [................XXXXXXX.XXXXXXXXXXX.XXXXXXX.................] duration:
988.380632ms trace:
https://jaeger.dev.storj.io/trace/4d0434aacaaec70b
FAIL (2025-12-11 00:01:19 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
.578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment
Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments...
Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments...
Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 988.39803ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...........XXX..............................................] duration:
714.780138ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-11 00:00:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 714.833554ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
22 out of 60 is failed (threshold=3) [...................XX..XXXXXX.XXXXXX.XXXXXXXX...............] duration:
409.823207ms trace:
https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8
FAIL (2025-12-11 00:00:44 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload
Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload
Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments...
Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false
Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment...
Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment.
Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment
Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments...
Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments...
Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 409.860808ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [.................XXXXXXX.XXXXXXXXXXX.XXXXXXX................] duration:
423.431814ms trace:
https://jaeger.dev.storj.io/trace/4d0434aacaaec70b
FAIL (2025-12-11 00:00:42 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
.578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment
Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments...
Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments...
Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 423.4461ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......X....X...X........X...X..............................] duration:
484.134209ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-11 00:00:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 484.15736ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [......XXX.X.................................................] duration:
516.118356ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-11 00:00:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 516.166455ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [........X....X...X........X...X.............................] duration:
518.810202ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-11 00:00:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 518.846418ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
22 out of 60 is failed (threshold=3) [....................XX..XXXXXX.XXXXXX.XXXXXXXX..............] duration:
659.712031ms trace:
https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8
FAIL (2025-12-11 00:00:14 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload
Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload
Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments...
Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false
Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment...
Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment.
Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment
Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments...
Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments...
Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 659.740961ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.......XXX.X................................................] duration:
518.789529ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-11 00:00:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 518.824894ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [............XXX.............................................] duration:
928.63486ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-11 00:00:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 928.74637ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.........X....X...X........X...X............................] duration:
472.140277ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:59:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 472.166818ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [..................XXXXXXX.XXXXXXXXXXX.XXXXXXX...............] duration:
518.998576ms trace:
https://jaeger.dev.storj.io/trace/4d0434aacaaec70b
FAIL (2025-12-10 23:59:43 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
.578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment
Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments...
Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments...
Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 519.028863ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
22 out of 60 is failed (threshold=3) [.....................XX..XXXXXX.XXXXXX.XXXXXXXX.............] duration:
392.190996ms trace:
https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8
FAIL (2025-12-10 23:59:30 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload
Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload
Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments...
Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false
Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment...
Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment.
Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment
Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments...
Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments...
Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 392.217845ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [........XXX.X...............................................] duration:
426.910628ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:59:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 426.925501ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.............XXX............................................] duration:
526.07524ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:59:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 526.106108ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..........X....X...X........X...X...........................] duration:
502.538275ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:59:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 502.573182ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [...................XXXXXXX.XXXXXXXXXXX.XXXXXXX..............] duration:
485.845744ms trace:
https://jaeger.dev.storj.io/trace/4d0434aacaaec70b
FAIL (2025-12-10 23:58:55 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
.578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment
Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments...
Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments...
Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 485.880632ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.........XXX.X..............................................] duration:
595.050423ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:58:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 595.149247ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
22 out of 60 is failed (threshold=3) [......................XX..XXXXXX.XXXXXX.XXXXXXXX............] duration:
359.719445ms trace:
https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8
FAIL (2025-12-10 23:58:28 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload
Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload
Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments...
Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false
Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment...
Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment.
Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment
Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments...
Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments...
Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 359.751056ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...........X....X...X........X...X..........................] duration:
467.825558ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:58:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 467.850517ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..........XXX.X.............................................] duration:
410.121001ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:58:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 410.200243ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [....................XXXXXXX.XXXXXXXXXXX.XXXXXXX.............] duration:
501.252164ms trace:
https://jaeger.dev.storj.io/trace/4d0434aacaaec70b
FAIL (2025-12-10 23:58:09 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
.578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment
Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments...
Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments...
Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 501.27685ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..............XXX...........................................] duration:
505.63525ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:58:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 505.663127ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
22 out of 60 is failed (threshold=3) [.......................XX..XXXXXX.XXXXXX.XXXXXXXX...........] duration:
409.499104ms trace:
https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8
FAIL (2025-12-10 23:58:03 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload
Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload
Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments...
Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false
Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment...
Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment.
Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment
Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments...
Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments...
Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 409.535748ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...........XXX.X............................................] duration:
406.861874ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:57:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 406.96783ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
22 out of 60 is failed (threshold=3) [........................XX..XXXXXX.XXXXXX.XXXXXXXX..........] duration:
432.98643ms trace:
https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8
FAIL (2025-12-10 23:57:46 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload
Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload
Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments...
Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false
Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment...
Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment.
Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment
Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments...
Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments...
Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 433.015387ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [............XXX.X...........................................] duration:
438.13704ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:57:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 438.173868ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [............X....X...X........X...X.........................] duration:
468.020807ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:57:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 468.046822ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [.....................XXXXXXX.XXXXXXXXXXX.XXXXXXX............] duration:
447.096868ms trace:
https://jaeger.dev.storj.io/trace/4d0434aacaaec70b
FAIL (2025-12-10 23:57:08 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
.578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment
Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments...
Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments...
Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 447.122829ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...............XXX..........................................] duration:
544.890376ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:56:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 544.911853ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [......................XXXXXXX.XXXXXXXXXXX.XXXXXXX...........] duration:
653.846787ms trace:
https://jaeger.dev.storj.io/trace/4d0434aacaaec70b
FAIL (2025-12-10 23:56:52 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
.578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment
Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments...
Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments...
Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 653.886969ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
22 out of 60 is failed (threshold=3) [.........................XX..XXXXXX.XXXXXX.XXXXXXXX.........] duration:
498.992172ms trace:
https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8
FAIL (2025-12-10 23:56:52 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload
Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload
Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments...
Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false
Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment...
Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment.
Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment
Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments...
Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments...
Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 499.07982ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|