Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 12 out of 60 is failed (threshold=3) [XXXXX.XXXXXXX...............................................] duration: 484.398323ms trace: https://jaeger.dev.storj.io/trace/4d0434aacaaec70b FAIL (2025-12-11 00:22:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments... Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments... Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 484.422502ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 480.559494ms trace: https://jaeger.dev.storj.io/trace/4f84f1680308bcc4 FAIL (2025-12-11 00:22:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 480.601142ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 16 out of 60 is failed (threshold=3) [XX.XXXXXX.XXXXXXXX..........................................] duration: 548.862506ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:21:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 548.893138ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 13 out of 60 is failed (threshold=3) [XXXXXX.XXXXXXX..............................................] duration: 553.222973ms trace: https://jaeger.dev.storj.io/trace/4d0434aacaaec70b FAIL (2025-12-11 00:21:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments... Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments... Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 553.253172ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 482.476308ms trace: https://jaeger.dev.storj.io/trace/4f84f1680308bcc4 FAIL (2025-12-11 00:21:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 482.504548ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 17 out of 60 is failed (threshold=3) [XXX.XXXXXX.XXXXXXXX.........................................] duration: 514.243148ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:21:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 514.295074ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 14 out of 60 is failed (threshold=3) [XXXXXXX.XXXXXXX.............................................] duration: 507.430797ms trace: https://jaeger.dev.storj.io/trace/4d0434aacaaec70b FAIL (2025-12-11 00:20:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments... Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments... Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 507.530811ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXXX.XXXXXX.XXXXXXXX........................................] duration: 423.570171ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:20:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 423.595511ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 448.61062ms trace: https://jaeger.dev.storj.io/trace/4f84f1680308bcc4 FAIL (2025-12-11 00:20:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 448.635424ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 15 out of 60 is failed (threshold=3) [XXXXXXXX.XXXXXXX............................................] duration: 531.894257ms trace: https://jaeger.dev.storj.io/trace/4d0434aacaaec70b FAIL (2025-12-11 00:20:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments... Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments... Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 531.926534ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 16 out of 60 is failed (threshold=3) [XXXXXXXXX.XXXXXXX...........................................] duration: 577.810281ms trace: https://jaeger.dev.storj.io/trace/4d0434aacaaec70b FAIL (2025-12-11 00:19:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments... Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments... Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 577.853897ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 19 out of 60 is failed (threshold=3) [XXXXX.XXXXXX.XXXXXXXX.......................................] duration: 397.480748ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:19:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 397.575634ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X...X.......................................................] duration: 460.850938ms trace: https://jaeger.dev.storj.io/trace/4f84f1680308bcc4 FAIL (2025-12-11 00:19:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 460.88493ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 20 out of 60 is failed (threshold=3) [XXXXXX.XXXXXX.XXXXXXXX......................................] duration: 386.65589ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:19:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 386.683287ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 17 out of 60 is failed (threshold=3) [XXXXXXXXXX.XXXXXXX..........................................] duration: 549.184806ms trace: https://jaeger.dev.storj.io/trace/4d0434aacaaec70b FAIL (2025-12-11 00:18:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments... Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments... Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 549.208048ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X...X......................................................] duration: 446.041915ms trace: https://jaeger.dev.storj.io/trace/4f84f1680308bcc4 FAIL (2025-12-11 00:18:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 446.075635ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 20 out of 60 is failed (threshold=3) [.XXXXXX.XXXXXX.XXXXXXXX.....................................] duration: 396.573231ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:18:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 396.610947ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X...X.....................................................] duration: 451.862613ms trace: https://jaeger.dev.storj.io/trace/4f84f1680308bcc4 FAIL (2025-12-11 00:18:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 451.891901ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 20 out of 60 is failed (threshold=3) [..XXXXXX.XXXXXX.XXXXXXXX....................................] duration: 439.982379ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:17:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 440.03633ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXXXXXXXXXX.XXXXXXX.........................................] duration: 406.772878ms trace: https://jaeger.dev.storj.io/trace/4d0434aacaaec70b FAIL (2025-12-11 00:17:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments... Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments... Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 406.802571ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X...X....................................................] duration: 474.778631ms trace: https://jaeger.dev.storj.io/trace/4f84f1680308bcc4 FAIL (2025-12-11 00:16:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 474.798242ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 21 out of 60 is failed (threshold=3) [X..XXXXXX.XXXXXX.XXXXXXXX...................................] duration: 381.07825ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:16:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 381.11146ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X...X...................................................] duration: 525.898135ms trace: https://jaeger.dev.storj.io/trace/4f84f1680308bcc4 FAIL (2025-12-11 00:16:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 525.923027ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 18 out of 60 is failed (threshold=3) [.XXXXXXXXXXX.XXXXXXX........................................] duration: 467.352068ms trace: https://jaeger.dev.storj.io/trace/4d0434aacaaec70b FAIL (2025-12-11 00:16:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments... Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments... Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 467.38151ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX..XXXXXX.XXXXXX.XXXXXXXX..................................] duration: 470.815147ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:16:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 470.847647ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XX..XXXXXX.XXXXXX.XXXXXXXX.................................] duration: 412.470885ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:15:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 412.501003ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.XXXXXXXXXXX.XXXXXXX.......................................] duration: 621.975131ms trace: https://jaeger.dev.storj.io/trace/4d0434aacaaec70b FAIL (2025-12-11 00:15:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments... Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments... Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 622.001333ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 20 out of 60 is failed (threshold=3) [XX.XXXXXXXXXXX.XXXXXXX......................................] duration: 565.428289ms trace: https://jaeger.dev.storj.io/trace/4d0434aacaaec70b FAIL (2025-12-11 00:15:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments... Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments... Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 565.502302ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X...X..................................................] duration: 507.003581ms trace: https://jaeger.dev.storj.io/trace/4f84f1680308bcc4 FAIL (2025-12-11 00:15:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 507.030407ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 22 out of 60 is failed (threshold=3) [..XX..XXXXXX.XXXXXX.XXXXXXXX................................] duration: 500.118185ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:15:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 500.145814ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X...X.................................................] duration: 507.19661ms trace: https://jaeger.dev.storj.io/trace/4f84f1680308bcc4 FAIL (2025-12-11 00:15:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 507.244037ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXX.XXXXXXXXXXX.XXXXXXX.....................................] duration: 509.433502ms trace: https://jaeger.dev.storj.io/trace/4d0434aacaaec70b FAIL (2025-12-11 00:14:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments... Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments... Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 509.46017ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X...X................................................] duration: 479.967293ms trace: https://jaeger.dev.storj.io/trace/4f84f1680308bcc4 FAIL (2025-12-11 00:14:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 479.993836ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 22 out of 60 is failed (threshold=3) [...XX..XXXXXX.XXXXXX.XXXXXXXX...............................] duration: 485.902103ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:14:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 485.930855ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXXX.XXXXXXXXXXX.XXXXXXX....................................] duration: 522.627641ms trace: https://jaeger.dev.storj.io/trace/4d0434aacaaec70b FAIL (2025-12-11 00:14:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments... Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments... Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 522.654537ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X...X...............................................] duration: 438.966062ms trace: https://jaeger.dev.storj.io/trace/4f84f1680308bcc4 FAIL (2025-12-11 00:14:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 439.105093ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 571.83971ms trace: https://jaeger.dev.storj.io/trace/3e967eec712b2e24 FAIL (2025-12-11 00:13:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 572.531327ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXXXX.XXXXXXXXXXX.XXXXXXX...................................] duration: 477.876153ms trace: https://jaeger.dev.storj.io/trace/4d0434aacaaec70b FAIL (2025-12-11 00:13:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments... Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments... Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 477.906075ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X........X...X..............................................] duration: 455.413058ms trace: https://jaeger.dev.storj.io/trace/4f84f1680308bcc4 FAIL (2025-12-11 00:13:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 455.439483ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 22 out of 60 is failed (threshold=3) [....XX..XXXXXX.XXXXXX.XXXXXXXX..............................] duration: 408.665189ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:12:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 408.697256ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 627.038669ms trace: https://jaeger.dev.storj.io/trace/3e967eec712b2e24 FAIL (2025-12-11 00:12:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 627.130409ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 22 out of 60 is failed (threshold=3) [.....XX..XXXXXX.XXXXXX.XXXXXXXX.............................] duration: 527.584519ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:12:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 527.636959ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXX.XXXXXXXXXXX.XXXXXXX..................................] duration: 445.159038ms trace: https://jaeger.dev.storj.io/trace/4d0434aacaaec70b FAIL (2025-12-11 00:12:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments... Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments... Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 445.178515ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X........X...X.............................................] duration: 467.632209ms trace: https://jaeger.dev.storj.io/trace/4f84f1680308bcc4 FAIL (2025-12-11 00:12:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 467.651133ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 22 out of 60 is failed (threshold=3) [......XX..XXXXXX.XXXXXX.XXXXXXXX............................] duration: 461.890018ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:12:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 461.942506ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXX.XXXXXXXXXXX.XXXXXXX.................................] duration: 451.441874ms trace: https://jaeger.dev.storj.io/trace/4d0434aacaaec70b FAIL (2025-12-11 00:12:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments... Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments... Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 451.470247ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 942.296422ms trace: https://jaeger.dev.storj.io/trace/3e967eec712b2e24 FAIL (2025-12-11 00:11:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 942.322426ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 22 out of 60 is failed (threshold=3) [.......XX..XXXXXX.XXXXXX.XXXXXXXX...........................] duration: 578.457886ms trace: https://jaeger.dev.storj.io/trace/22c19ffca5e1b3b8 FAIL (2025-12-11 00:11:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Dec 10 23:48:30.013175102 (128h20m30.840883494s) [upload=133452]: Starting upload Dec 10 23:48:30.013246055 (128h20m30.840954455s) [upload=133452]: Done starting upload Dec 10 23:48:30.013314524 (128h20m30.841022916s) [upload=133452]: Uploading segments... Dec 10 23:48:30.013569004 (128h20m30.841277396s) [upload=133452]: Got next segment. Inline: false Dec 10 23:48:30.013613480 (128h20m30.841321871s) [upload=133452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:48:30.013624259 (128h20m30.841332653s) [upload=133452]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:48:30.042566737 (128h20m30.870275138s) [seg_pos={0 0}, upload=133452]: Begin upload segment... Dec 10 23:48:30.042647212 (128h20m30.870355612s) [seg_pos={0 0}, upload=133452]: Done begin upload segment. Dec 10 23:48:30.042667181 (128h20m30.870375563s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042695665 (128h20m30.870404067s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.042768067 (128h20m30.870476463s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043390495 (128h20m30.871098897s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.043807571 (128h20m30.871515966s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.044475122 (128h20m30.872183514s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045070309 (128h20m30.872778706s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.045468993 (128h20m30.873177387s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.047663619 (128h20m30.875371997s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.048482268 (128h20m30.876190662s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049130168 (128h20m30.876838565s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.049628215 (128h20m30.877336613s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.050501554 (128h20m30.878209951s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051102761 (128h20m30.878811158s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051581383 (128h20m30.879289765s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.051988364 (128h20m30.87969676s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052364021 (128h20m30.880072416s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.052767608 (128h20m30.880476008s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Uploading piece... Dec 10 23:48:30.116894175 (128h20m30.944602553s) [upload=133452]: Next returned nil segment Dec 10 23:48:30.116907727 (128h20m30.944616104s) [upload=133452]: Waiting for error group managing segments... Dec 10 23:48:30.121410783 (128h20m30.949119177s) [noise=true , address=45.91.116.14:4200 , piece_id=YDVZOKTUROJH6UTOZGJCLXSJL2IRF3C5V4T6OVA34HUMGLV6FMYA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.137755075 (128h20m30.96546347s) [noise=true , address=45.91.116.11:4144 , piece_id=DDAMIAP74H7SFB2BHJB47RLM6AJJAHE64OCTTWPOFU3ZDLLUN5RQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.138917870 (128h20m30.966626262s) [noise=true , address=45.91.116.8:4062 , piece_id=URHMWBMHW4NHUUSMXRYC465GCPZASZUYUQUS3WTXIM5ZKM2SMJ6A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141562850 (128h20m30.969271245s) [noise=true , address=145.79.227.49:4344 , piece_id=JTXFQBWGXJOJTHVOKFYUFWPGACHAYX73I65IJB6CESDSQWNB2W2A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.141651483 (128h20m30.969359876s) [noise=true , address=45.91.116.5:4300 , piece_id=NBNMMDLSXIMJSFILMEMNWIKYUM6VKOVKCHOG33LCOGR6EK76H77Q, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.142548753 (128h20m30.970257154s) [noise=true , address=45.91.116.10:4114 , piece_id=YEWKDEMDLRL5YULCEJI7PV27NKXIZEWTLSN4TFQ5ZGFLK37MLBRQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.151002606 (128h20m30.978710997s) [noise=true , address=145.79.227.5:4237 , piece_id=TFDMTWHBI67AT2XRTQPJAT2ZCK3DJ5QEGMOQ4TKAUWU54UMEG2UA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.229287937 (128h20m31.056996329s) [noise=true , address=45.91.116.13:4170 , piece_id=PHAVUYWW3TWNCPOO5FPUSTYXXI37T66NQ4MOM3GO2S3GGY5CW25A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.380881869 (128h20m31.208590269s) [noise=true , address=145.79.227.40:4062 , piece_id=S2I56TSC4V23QRYDKK2QGXW7UYP2KBLNJ7FW5NDGVCYGASV4MKMQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:30.644613768 (128h20m31.47232216s) [noise=true , address=64.49.13.155:4924 , piece_id=XWBQ44PIR2CEG7H7GWAR3VEADGZELEOZLLF3XROUI577BA5VSYQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.075436652 (128h20m31.903145063s) [noise=true , address=145.79.227.44:4227 , piece_id=UQRXBZVAANW4G5OJDH32TQY7G6TLSLBPFKXLVHS7V3JEVDYA3BBA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:48:31.179221360 (128h20m32.006929751s) [noise=true , address=145.79.227.30:4790 , piece_id=LASFBK7FILPTUGBE2XLED4W2JVQK3L5BM7YHCX64RBDZSVMUUTIA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: <nil> Dec 10 23:49:29.976420551 (128h21m30.804128946s) [noise=true , address=77.90.31.10:4542 , piece_id=OCNSNPJ77AFO4KOWNDIWHMF4RW46PEFVALUN63OKM3UGAUGEBUBQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1T8wyLvnYL2oRtQUhw8FhVFZ98Q5DhyRv1EnkCNAbW8TqcAjJc, address:77.90.31.10:4542): protocol: rpc: dial tcp 77.90.31.10:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976794812 (128h21m30.804503206s) [noise=true , address=77.90.31.19:4868 , piece_id=FN5T2AS56HP5ZWDEI5G4ROB33OLYPAOQ4BPAEWGG3ISGF2WCPLQQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976854126 (128h21m30.80456252s) [noise=true , address=77.90.31.14:4681 , piece_id=VQB2XIC6RUWZNLK2PMSGRBPK5WSGS4TJGEFNXUXA53T4ZANNZT4A, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: rpc: dial tcp 77.90.31.14:4681: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.976990445 (128h21m30.804698836s) [noise=true , address=77.90.31.13:4654 , piece_id=6DJLQ2CT4XNWPBLTDIKDEUCHQSFPGEWVFX5ON5KE6PQT3WXWBVVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: write tcp 159.223.143.39:56540->77.90.31.13:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977037908 (128h21m30.804746302s) [noise=true , address=77.90.31.16:4761 , piece_id=U6ZYNY6YTHGCSAD7MN75VL27KB3YI6SVWEVEU4DSR3GTOJSBIKVQ, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: write tcp 159.223.143.39:47512->77.90.31.16:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977188955 (128h21m30.804897349s) [noise=true , address=77.90.31.18:4814 , piece_id=ILBMJFZVSEJF5JNZTSWZSKVSDL3R7XADGH6I6IHDAG44ZX574VAA, seg_pos={0 0}, upload=133452]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: rpc: dial tcp 77.90.31.18:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:49:29.977212561 (128h21m30.804920953s) [seg_pos={0 0}, upload=133452]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:49:30.006871616 (128h21m30.834580019s) [upload=133452]: Done uploading segments... Dec 10 23:49:30.006929034 (128h21m30.834637434s) [upload=133452]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 578.490391ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X........X...X............................................] duration: 495.061706ms trace: https://jaeger.dev.storj.io/trace/4f84f1680308bcc4 FAIL (2025-12-11 00:11:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 495.094032ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 25 out of 60 is failed (threshold=3) [.XXXXXXX.XXXXXXXXXXX.XXXXXXX................................] duration: 577.297107ms trace: https://jaeger.dev.storj.io/trace/4d0434aacaaec70b FAIL (2025-12-11 00:11:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .578894079s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.751694854 (128h21m45.579403251s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752195427 (128h21m45.579903818s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.752754460 (128h21m45.580462854s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753213040 (128h21m45.580921441s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.753639700 (128h21m45.581348094s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.754112593 (128h21m45.581820987s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755287696 (128h21m45.582996091s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.755511942 (128h21m45.583220339s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756944466 (128h21m45.584652866s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.756970854 (128h21m45.584679248s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.758465348 (128h21m45.586173729s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.759867992 (128h21m45.587576389s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.761336461 (128h21m45.589044862s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.763147526 (128h21m45.590855925s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.764447729 (128h21m45.592156129s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.765735591 (128h21m45.593443988s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Uploading piece... Dec 10 23:49:44.813006026 (128h21m45.640714417s) [upload=133471]: Next returned nil segment Dec 10 23:49:44.813024228 (128h21m45.64073262s) [upload=133471]: Waiting for error group managing segments... Dec 10 23:49:44.817385006 (128h21m45.645093408s) [noise=true , address=45.91.116.9:4288 , piece_id=ALAEC5HWCQAXMZF3VPR4XAH3JKG2VT4NGV54T3JVURSM5K4WXP5A, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818156983 (128h21m45.64586538s) [noise=true , address=45.91.116.4:4958 , piece_id=HU7XUROFANT5C6CI6F5AQEQ3XC7STOVWNI73M47HVVJND3BR2B2Q, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.818785574 (128h21m45.646493973s) [noise=true , address=45.91.116.5:4314 , piece_id=BJ62SXYVCBXEDV67VESM4ELF4GLY3GKHKFIV3QPPWPXHPHHV37MA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819301709 (128h21m45.647010102s) [noise=true , address=145.79.227.45:4262 , piece_id=6MFNTK57MNLEGNM4SFB7EWDOFG2FQJQUTPFX55X77CHDWKD46RWQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.819488167 (128h21m45.647196567s) [noise=true , address=145.79.227.27:4698 , piece_id=UMSQBHFKS5L3IDR42SSK7JXXM4SFL5FSRLCWP4RHTUH6HBAONPOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.831211867 (128h21m45.658920263s) [noise=true , address=45.91.116.14:4219 , piece_id=PJ3DL4RC5GYTTFY5Q3IKZIIP2QASIKB45AMAFKD3BNQ7SRURRQLQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:44.932120147 (128h21m45.759828541s) [noise=true , address=5.231.73.4:4024 , piece_id=3SORCJAOP3MYXXPDCXI6MDCI5V43U4ZJA7MVO4A4M2E2VAHNRJDA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.859417997 (128h21m46.687126388s) [noise=true , address=145.79.227.16:4917 , piece_id=ANQN5TYCPHD6IIRWINWSLIA7IPDFBKTBURCEO2ZCH3BOO7ZYVNAQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:45.906927127 (128h21m46.734635506s) [noise=true , address=145.79.227.30:4765 , piece_id=TOACIQYCYTLKJ5EZPNHTKL37VHX2ATQA3Q5M5NFSYPYX45JXJVIA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:49:54.577349480 (128h21m55.405057874s) [noise=true , address=64.49.13.169:4706 , piece_id=2DBGQYASJIHWANJEFAV65G7NLHX5YIOXIB44EUKUWSHD3SEGICGA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: <nil> Dec 10 23:50:44.675786234 (128h22m45.503494637s) [noise=true , address=77.90.31.19:4848 , piece_id=3RRXO4TCXXUWGOCC3XPSCQGBQRJHV3HUN4VXSOB5TETD7UQ5G7BA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: write tcp 159.223.143.39:49878->77.90.31.19:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676071397 (128h22m45.503780038s) [noise=true , address=77.90.31.11:4564 , piece_id=ETFMGZ5I5HN6FSXVR7AV42LGYFNOXDH4TQEILKUHCXHWGJI7V6TQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: write tcp 159.223.143.39:38596->77.90.31.11:4564: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.676287191 (128h22m45.503995582s) [noise=true , address=77.90.31.13:4646 , piece_id=QMC44KSD7WST5TSP4CB3C6XJVHHAP5UPH4MKTJWWCT3YJ7QV4XSQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, address:77.90.31.13:4646): protocol: write tcp 159.223.143.39:52684->77.90.31.13:4646: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677301970 (128h22m45.505010368s) [noise=true , address=77.90.31.16:4762 , piece_id=WPLDTRNFUBE6R4ZMVXI44UWOXHM7H4ANQCFF2UVR6FHQ2FM4QNUA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677360272 (128h22m45.505068679s) [noise=true , address=77.90.31.15:4709 , piece_id=H4T4XS5KNVCB7N2TVTAD3HXEHJW5NOAJSFX77GDZ7YFCRE535EJA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 159.223.143.39:37288->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677518090 (128h22m45.505226496s) [noise=true , address=77.90.31.10:4535 , piece_id=PPWUFKPTY2OEAMEBC67LUXWUBX2M46GIWXSKNUB6O36OLE4PUILA, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, address:77.90.31.10:4535): protocol: write tcp 159.223.143.39:36798->77.90.31.10:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.677579077 (128h22m45.50528748s) [noise=true , address=77.90.31.16:4756 , piece_id=OYAPIMHZPUYUHULHIUB2LJAARO5K57M7VDZ5C2URHBZZDVDUICOQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, address:77.90.31.16:4756): protocol: write tcp 159.223.143.39:56606->77.90.31.16:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:50:44.678021187 (128h22m45.505729579s) [noise=true , address=77.90.31.18:4837 , piece_id=M4IXXEZ4WWRXB6VBKWQAUXWMLI6KGAWQ4JPAHQRGZKJH5OSBQ3IQ, seg_pos={0 0}, upload=133471]: Done uploading piece. err: ecclient: upload failed (node:12YfSnynVtsWn1XtXHGeik1vUgqzRtArmEJJrWcXd23m2yUTS8j, address:77.90.31.18:4837): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:50:44.678061800 (128h22m45.505770195s) [seg_pos={0 0}, upload=133471]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:50:44.714103824 (128h22m45.541812216s) [upload=133471]: Done uploading segments... Dec 10 23:50:44.714159127 (128h22m45.541867527s) [upload=133471]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 577.329025ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)