Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.............XXX.X..........................................] duration:
426.959795ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:56:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 426.989658ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
22 out of 60 is failed (threshold=3) [..........................XX..XXXXXX.XXXXXX.XXXXXXXX........] duration:
420.577108ms trace:
https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8
FAIL (2025-12-10 23:56:36 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload
Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload
Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments...
Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false
Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment...
Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment.
Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment
Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments...
Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments...
Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 420.692321ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [.......................XXXXXXX.XXXXXXXXXXX.XXXXXXX..........] duration:
530.869485ms trace:
https://jaeger.dev.storj.io/trace/4d0434aacaaec70b
FAIL (2025-12-10 23:56:22 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
.578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment
Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments...
Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments...
Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 530.918659ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..............XXX.X.........................................] duration:
434.494142ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:56:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 434.962337ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.............X....X...X........X...X........................] duration:
492.293954ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:56:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 492.322703ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
22 out of 60 is failed (threshold=3) [...........................XX..XXXXXX.XXXXXX.XXXXXXXX.......] duration:
375.235241ms trace:
https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8
FAIL (2025-12-10 23:55:58 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload
Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload
Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments...
Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false
Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment...
Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment.
Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment
Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments...
Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments...
Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 375.26465ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [................XXX.........................................] duration:
614.903317ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:55:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 615.00693ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [........................XXXXXXX.XXXXXXXXXXX.XXXXXXX.........] duration:
391.997619ms trace:
https://jaeger.dev.storj.io/trace/4d0434aacaaec70b
FAIL (2025-12-10 23:55:49 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
.578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment
Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments...
Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments...
Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 392.016211ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [.........................XXXXXXX.XXXXXXXXXXX.XXXXXXX........] duration:
593.067773ms trace:
https://jaeger.dev.storj.io/trace/4d0434aacaaec70b
FAIL (2025-12-10 23:55:34 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
.578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment
Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments...
Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments...
Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 593.088627ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..............X....X...X........X...X.......................] duration:
469.40726ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:55:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 469.428506ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...............XXX.X........................................] duration:
456.748285ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:55:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 456.777959ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
22 out of 60 is failed (threshold=3) [............................XX..XXXXXX.XXXXXX.XXXXXXXX......] duration:
567.054243ms trace:
https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8
FAIL (2025-12-10 23:55:13 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload
Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload
Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments...
Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false
Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment...
Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment.
Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment
Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments...
Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments...
Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 567.081175ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.................XXX........................................] duration:
671.075053ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:55:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 671.172589ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [................XXX.X.......................................] duration:
393.370806ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:54:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 393.400573ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [..........................XXXXXXX.XXXXXXXXXXX.XXXXXXX.......] duration:
480.583688ms trace:
https://jaeger.dev.storj.io/trace/4d0434aacaaec70b
FAIL (2025-12-10 23:54:45 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
.578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment
Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments...
Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments...
Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 480.678253ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...............X....X...X........X...X......................] duration:
466.562911ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:54:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 466.577057ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.................XXX.X......................................] duration:
415.036915ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:54:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 415.066487ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [...........................XXXXXXX.XXXXXXXXXXX.XXXXXXX......] duration:
367.559592ms trace:
https://jaeger.dev.storj.io/trace/4d0434aacaaec70b
FAIL (2025-12-10 23:54:19 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
.578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment
Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments...
Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments...
Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 367.587915ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
22 out of 60 is failed (threshold=3) [.............................XX..XXXXXX.XXXXXX.XXXXXXXX.....] duration:
430.92303ms trace:
https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8
FAIL (2025-12-10 23:54:00 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload
Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload
Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments...
Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false
Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment...
Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment.
Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment
Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments...
Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments...
Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 430.949955ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..................XXX.......................................] duration:
553.076411ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:54:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 553.10734ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [............................XXXXXXX.XXXXXXXXXXX.XXXXXXX.....] duration:
427.070168ms trace:
https://jaeger.dev.storj.io/trace/4d0434aacaaec70b
FAIL (2025-12-10 23:53:53 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
.578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment
Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments...
Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments...
Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 427.09614ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [................X....X...X........X...X.....................] duration:
472.789302ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:53:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 472.898756ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..................XXX.X.....................................] duration:
419.890126ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:53:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 419.912806ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.................X....X...X........X...X....................] duration:
462.661543ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:53:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 462.688047ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...................XXX......................................] duration:
559.159527ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:53:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 559.184604ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...................XXX.X....................................] duration:
452.735115ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:53:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 452.768617ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
22 out of 60 is failed (threshold=3) [..............................XX..XXXXXX.XXXXXX.XXXXXXXX....] duration:
358.906108ms trace:
https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8
FAIL (2025-12-10 23:52:54 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload
Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload
Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments...
Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false
Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment...
Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment.
Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment
Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments...
Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments...
Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 358.934592ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [.............................XXXXXXX.XXXXXXXXXXX.XXXXXXX....] duration:
957.037239ms trace:
https://jaeger.dev.storj.io/trace/4d0434aacaaec70b
FAIL (2025-12-10 23:52:39 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
.578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment
Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments...
Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments...
Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 957.062145ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [....................XXX.....................................] duration:
530.36155ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:52:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 530.417296ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..................X....X...X........X...X...................] duration:
468.024856ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:52:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 468.050071ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [..............................XXXXXXX.XXXXXXXXXXX.XXXXXXX...] duration:
406.214586ms trace:
https://jaeger.dev.storj.io/trace/4d0434aacaaec70b
FAIL (2025-12-10 23:52:10 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
.578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment
Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments...
Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments...
Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 406.252656ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [....................XXX.X...................................] duration:
450.55171ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:52:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 450.582967ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
22 out of 60 is failed (threshold=3) [...............................XX..XXXXXX.XXXXXX.XXXXXXXX...] duration:
430.082072ms trace:
https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8
FAIL (2025-12-10 23:51:56 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload
Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload
Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments...
Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false
Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment...
Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment.
Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment
Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments...
Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments...
Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 430.120067ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [...............................XXXXXXX.XXXXXXXXXXX.XXXXXXX..] duration:
506.613597ms trace:
https://jaeger.dev.storj.io/trace/4d0434aacaaec70b
FAIL (2025-12-10 23:51:35 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
.578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment
Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments...
Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments...
Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 506.645034ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.....................XXX.X..................................] duration:
586.928863ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:51:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 587.021168ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...................X....X...X........X...X..................] duration:
491.174534ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:51:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 491.196324ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.....................XXX....................................] duration:
593.333884ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:51:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 593.3673ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [......................XXX.X.................................] duration:
457.827756ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:50:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 457.903468ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [................................XXXXXXX.XXXXXXXXXXX.XXXXXXX.] duration:
607.748292ms trace:
https://jaeger.dev.storj.io/trace/4d0434aacaaec70b
FAIL (2025-12-10 23:50:45 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
.578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment
Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments...
Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments...
Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 607.768709ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [.................................XXXXXXX.XXXXXXXXXXX.XXXXXXX] duration:
1m0.042675898s trace:
https://jaeger.dev.storj.io/trace/4d0434aacaaec70b
FAIL (2025-12-10 23:50:44 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
.578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece...
Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment
Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments...
Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil>
Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments...
Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 1m0.042689859s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [......................XXX...................................] duration:
589.531555ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:50:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 589.867627ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
22 out of 60 is failed (threshold=3) [................................XX..XXXXXX.XXXXXX.XXXXXXXX..] duration:
466.27031ms trace:
https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8
FAIL (2025-12-10 23:50:42 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload
Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload
Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments...
Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false
Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment...
Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment.
Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment
Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments...
Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments...
Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 466.371653ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [....................X....X...X........X...X.................] duration:
456.362846ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:50:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 456.661417ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.......................XXX.X................................] duration:
428.090685ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:50:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 428.124214ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.......................XXX..................................] duration:
574.116286ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:49:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 574.177168ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [........................XXX.X...............................] duration:
420.024518ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:49:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 420.053287ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
24 out of 60 is failed (threshold=3) [..................................XXXXXXX.XXXXXXXXXXX.XXXXXX] duration:
1m0.026113633s trace:
https://jaeger.dev.storj.io/trace/5320d151c38e9ed
FAIL (2025-12-10 23:49:44 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
upload segment...
Dec 10 23:48:44.647849449 (128h20m45.475557834s) [seg_pos={0 0}, upload=133456]: Done begin upload segment.
Dec 10 23:48:44.647897278 (128h20m45.475605676s) [noise=true , address=45.91.116.9:4267 , piece_id=FTSDEOUW26L3YYP3TAMHWWK3KIYB2VMOZMZOEXZZLH7PSRTP22CQ, seg_pos={0 0}, upload=133456]: Uploading piece...
Dec 10 23:48:44.647916750 (128h20m45.475625188s) [noise=true , address=145.79.227.46:4625 , piece_id=7MW66JPJOHJ2JZFFDXMK76AK7W4FJXFXRDBSGGU67BI3RW2E5MYQ, seg_pos={0 0}, upload=133456]: Uploading piece...
Dec 10 23:48:44.647981935 (128h20m45.47569032s) [noise=true , address=145.79.227.45:4236 , piece_id=G3B3M2XGUTEYNUAHKCP4XAMKM2HE47Q3AFR7JE5SJLKTMKVALZEA, seg_pos={0 0}, upload=133456]: Uploading piece...
Dec 10 23:48:44.648426036 (128h20m45.4761345s) [noise=true , address=45.91.116.11:4134 , piece_id=PL5CUEA43OXSY3J4GJKEIIIKDPPDZC4GX5EKRJX6KWYULO55F6ZA, seg_pos={0 0}, upload=133456]: Uploading piece...
Dec 10 23:48:44.649170312 (128h20m45.476878714s) [noise=true , address=64.49.13.138:4490 , piece_id=UL4MK7U2SQRSR3RFQYBZRFA55AQTRB6EW74ARXMJRBLYUSULSMLQ, seg_pos={0 0}, upload=133456]: Uploading piece...
Dec 10 23:48:44.649547747 (128h20m45.47725615s) [noise=true , address=77.90.31.16:4736 , piece_id=OXSQP3HICRJ5J3PXMDLZTCQQS5FJ3QMBLFCNS2IDRBDQPB43EZ3A, seg_pos={0 0}, upload=133456]: Uploading piece...
Dec 10 23:48:44.649702458 (128h20m45.47741085s) [noise=true , address=77.90.31.15:4718 , piece_id=VGMC7ESCM4HULYPSYAW6DTX5BUFBZK4MZKY7W446RJUGB3YJRM6A, seg_pos={0 0}, upload=133456]: Uploading piece...
Dec 10 23:48:44.650149944 (128h20m45.47785834s) [noise=true , address=145.79.227.30:4770 , piece_id=V436U3QJZPBKE7LCDV53KBAZ5RFHT5G5ZRAAKHWDV33S3IU6KRJA, seg_pos={0 0}, upload=133456]: Uploading piece...
Dec 10 23:48:44.650650732 (128h20m45.478359123s) [noise=true , address=145.79.227.5:4246 , piece_id=AAPPZTQTPTI2DF33WIHQBRX72TI5QZD2SDM6K4IAYKCMZQSEXSNA, seg_pos={0 0}, upload=133456]: Uploading piece...
Dec 10 23:48:44.651064955 (128h20m45.478773352s) [noise=true , address=45.91.116.11:4143 , piece_id=M7QNFKVRTIITY7JULAUCAKJFDW533EXHP6PKJQPBSKTGRHENU2SA, seg_pos={0 0}, upload=133456]: Uploading piece...
Dec 10 23:48:44.651576372 (128h20m45.479284773s) [noise=true , address=77.90.31.11:4574 , piece_id=UPL4CUVJKVSKA6YQ4FZ2JRG2GZVZ47DZAGRMKNWJSR5VAPWAYBOA, seg_pos={0 0}, upload=133456]: Uploading piece...
Dec 10 23:48:44.651726348 (128h20m45.479434742s) [noise=true , address=77.90.31.21:4907 , piece_id=DOYKXNRDOOR6J4SPDFTT2X5AVLZBTTPFVVA3QMX7DUMH4LCIQ46A, seg_pos={0 0}, upload=133456]: Uploading piece...
Dec 10 23:48:44.652103107 (128h20m45.479811497s) [noise=true , address=77.90.31.12:4600 , piece_id=GKDGQZCZD2X6VBSRHFTPPE2TAKZCK5FGNWIITXDILPXMIVAYIFYA, seg_pos={0 0}, upload=133456]: Uploading piece...
Dec 10 23:48:44.652439221 (128h20m45.480147612s) [noise=true , address=145.79.227.32:4837 , piece_id=FMYUEPJ7G6K4JNMVWPJUFQSKMYQR5S7EEUX4FOPGMKGWKJKEMJXQ, seg_pos={0 0}, upload=133456]: Uploading piece...
Dec 10 23:48:44.652849924 (128h20m45.480558314s) [noise=true , address=145.79.227.27:4697 , piece_id=NTB65VUEDA5KN756ORMSKVKVMZY7PF4K3ODA757PY37LTAWCE6SA, seg_pos={0 0}, upload=133456]: Uploading piece...
Dec 10 23:48:44.653290441 (128h20m45.480998836s) [noise=true , address=45.91.116.13:4171 , piece_id=JSWO6N6QMZNVEYEYZC2BIVNV7AE4FU2H5GV3OH65QFKPZWW7U36Q, seg_pos={0 0}, upload=133456]: Uploading piece...
Dec 10 23:48:44.653490042 (128h20m45.481198442s) [noise=true , address=77.90.31.11:4584 , piece_id=2VOQFNGBQIKSCIH56MI3O47RQXAKESJNOYEP7TRFEL2AFUG5B43Q, seg_pos={0 0}, upload=133456]: Uploading piece...
Dec 10 23:48:44.653983721 (128h20m45.481692119s) [noise=true , address=77.90.31.19:4854 , piece_id=KNKNW2N45RHARYP6S5GW4OQQHVH2INQ5CE4AQWAOMKNIG5TDMXKA, seg_pos={0 0}, upload=133456]: Uploading piece...
Dec 10 23:48:44.724050397 (128h20m45.55175879s) [upload=133456]: Next returned nil segment
Dec 10 23:48:44.724063588 (128h20m45.551771969s) [upload=133456]: Waiting for error group managing segments...
Dec 10 23:48:44.731878875 (128h20m45.559587272s) [noise=true , address=145.79.227.27:4697 , piece_id=NTB65VUEDA5KN756ORMSKVKVMZY7PF4K3ODA757PY37LTAWCE6SA, seg_pos={0 0}, upload=133456]: Done uploading piece. err: <nil>
Dec 10 23:48:44.737670610 (128h20m45.565379004s) [noise=true , address=45.91.116.11:4134 , piece_id=PL5CUEA43OXSY3J4GJKEIIIKDPPDZC4GX5EKRJX6KWYULO55F6ZA, seg_pos={0 0}, upload=133456]: Done uploading piece. err: <nil>
Dec 10 23:48:44.739620874 (128h20m45.567329274s) [noise=true , address=145.79.227.46:4625 , piece_id=7MW66JPJOHJ2JZFFDXMK76AK7W4FJXFXRDBSGGU67BI3RW2E5MYQ, seg_pos={0 0}, upload=133456]: Done uploading piece. err: <nil>
Dec 10 23:48:44.743613668 (128h20m45.57132206s) [noise=true , address=45.91.116.13:4171 , piece_id=JSWO6N6QMZNVEYEYZC2BIVNV7AE4FU2H5GV3OH65QFKPZWW7U36Q, seg_pos={0 0}, upload=133456]: Done uploading piece. err: <nil>
Dec 10 23:48:44.744695651 (128h20m45.572404043s) [noise=true , address=45.91.116.11:4143 , piece_id=M7QNFKVRTIITY7JULAUCAKJFDW533EXHP6PKJQPBSKTGRHENU2SA, seg_pos={0 0}, upload=133456]: Done uploading piece. err: <nil>
Dec 10 23:48:44.747770330 (128h20m45.57547871s) [noise=true , address=45.91.116.9:4267 , piece_id=FTSDEOUW26L3YYP3TAMHWWK3KIYB2VMOZMZOEXZZLH7PSRTP22CQ, seg_pos={0 0}, upload=133456]: Done uploading piece. err: <nil>
Dec 10 23:48:44.748262979 (128h20m45.575971364s) [noise=true , address=145.79.227.45:4236 , piece_id=G3B3M2XGUTEYNUAHKCP4XAMKM2HE47Q3AFR7JE5SJLKTMKVALZEA, seg_pos={0 0}, upload=133456]: Done uploading piece. err: <nil>
Dec 10 23:48:44.764381771 (128h20m45.592090163s) [noise=true , address=145.79.227.5:4246 , piece_id=AAPPZTQTPTI2DF33WIHQBRX72TI5QZD2SDM6K4IAYKCMZQSEXSNA, seg_pos={0 0}, upload=133456]: Done uploading piece. err: <nil>
Dec 10 23:48:44.784329224 (128h20m45.612037616s) [noise=true , address=145.79.227.32:4837 , piece_id=FMYUEPJ7G6K4JNMVWPJUFQSKMYQR5S7EEUX4FOPGMKGWKJKEMJXQ, seg_pos={0 0}, upload=133456]: Done uploading piece. err: <nil>
Dec 10 23:48:45.251686494 (128h20m46.079394892s) [noise=true , address=64.49.13.138:4490 , piece_id=UL4MK7U2SQRSR3RFQYBZRFA55AQTRB6EW74ARXMJRBLYUSULSMLQ, seg_pos={0 0}, upload=133456]: Done uploading piece. err: <nil>
Dec 10 23:48:45.699841798 (128h20m46.527550193s) [noise=true , address=145.79.227.30:4770 , piece_id=V436U3QJZPBKE7LCDV53KBAZ5RFHT5G5ZRAAKHWDV33S3IU6KRJA, seg_pos={0 0}, upload=133456]: Done uploading piece. err: <nil>
Dec 10 23:49:44.576230307 (128h21m45.403938704s) [noise=true , address=77.90.31.21:4907 , piece_id=DOYKXNRDOOR6J4SPDFTT2X5AVLZBTTPFVVA3QMX7DUMH4LCIQ46A, seg_pos={0 0}, upload=133456]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: rpc: dial tcp 77.90.31.21:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:44.576529704 (128h21m45.404238102s) [noise=true , address=77.90.31.15:4718 , piece_id=VGMC7ESCM4HULYPSYAW6DTX5BUFBZK4MZKY7W446RJUGB3YJRM6A, seg_pos={0 0}, upload=133456]: Done uploading piece. err: ecclient: upload failed (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC, address:77.90.31.15:4718): protocol: rpc: dial tcp 77.90.31.15:4718: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:44.576988403 (128h21m45.4046968s) [noise=true , address=77.90.31.11:4574 , piece_id=UPL4CUVJKVSKA6YQ4FZ2JRG2GZVZ47DZAGRMKNWJSR5VAPWAYBOA, seg_pos={0 0}, upload=133456]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:44.577262754 (128h21m45.404971153s) [noise=true , address=77.90.31.12:4600 , piece_id=GKDGQZCZD2X6VBSRHFTPPE2TAKZCK5FGNWIITXDILPXMIVAYIFYA, seg_pos={0 0}, upload=133456]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:44.577539868 (128h21m45.405248264s) [noise=true , address=77.90.31.11:4584 , piece_id=2VOQFNGBQIKSCIH56MI3O47RQXAKESJNOYEP7TRFEL2AFUG5B43Q, seg_pos={0 0}, upload=133456]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:44.577812455 (128h21m45.405520858s) [noise=true , address=77.90.31.19:4854 , piece_id=KNKNW2N45RHARYP6S5GW4OQQHVH2INQ5CE4AQWAOMKNIG5TDMXKA, seg_pos={0 0}, upload=133456]: Done uploading piece. err: ecclient: upload failed (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE, address:77.90.31.19:4854): protocol: rpc: dial tcp 77.90.31.19:4854: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:44.578219209 (128h21m45.405927603s) [noise=true , address=77.90.31.16:4736 , piece_id=OXSQP3HICRJ5J3PXMDLZTCQQS5FJ3QMBLFCNS2IDRBDQPB43EZ3A, seg_pos={0 0}, upload=133456]: Done uploading piece. err: ecclient: upload failed (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, address:77.90.31.16:4736): protocol: write tcp 159.223.143.39:41466->77.90.31.16:4736: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:44.578249757 (128h21m45.405958142s) [seg_pos={0 0}, upload=133456]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:49:44.600053193 (128h21m45.427761587s) [upload=133456]: Done uploading segments...
Dec 10 23:49:44.600097648 (128h21m45.427806042s) [upload=133456]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 1m0.026176709s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....................X....X...X........X...X................] duration:
497.878512ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:49:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 497.90569ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
22 out of 60 is failed (threshold=3) [.................................XX..XXXXXX.XXXXXX.XXXXXXXX.] duration:
1.295775412s trace:
https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8
FAIL (2025-12-10 23:49:31 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload
Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload
Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments...
Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false
Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment...
Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment.
Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment
Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments...
Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments...
Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1.295797549s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
22 out of 60 is failed (threshold=3) [..................................XX..XXXXXX.XXXXXX.XXXXXXXX] duration:
1m0.03289457s trace:
https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8
FAIL (2025-12-10 23:49:30 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload
Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload
Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments...
Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false
Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment...
Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment.
Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece...
Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment
Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments...
Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil>
Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments...
Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1m0.032999528s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|