Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................................XX........] duration: 415.961011ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:14:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 415.981333ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 573.537077ms trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:14:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 573.571097ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 414.26727ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:14:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 414.292997ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................................................XX.......] duration: 403.346422ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:14:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 403.36835ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 612.387634ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:14:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 612.401163ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 458.236507ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:13:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 458.280495ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 405.283744ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:13:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 405.308572ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 560.668026ms trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:13:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 560.697017ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................................................XX......] duration: 457.841624ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:12:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 457.976572ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 510.002218ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:12:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 510.029238ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 403.54106ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:12:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 403.569767ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 542.390394ms trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:12:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 542.433985ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 520.69753ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:11:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 520.719567ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 1.249207334s trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:11:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1.249234003s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................................................XX.....] duration: 425.994334ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:11:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 426.77761ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 530.030158ms trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:11:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 530.059691ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 457.808753ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:11:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 457.857013ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 528.586711ms trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:11:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 528.611858ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 483.793632ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:10:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 483.832315ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 346.55605ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:10:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 346.584364ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................................................XX....] duration: 425.806859ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:10:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 425.829185ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 404.935886ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:10:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 404.967339ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................................................XX...] duration: 442.354546ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:10:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 442.384801ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 641.567393ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:10:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 641.593149ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 518.75651ms trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:09:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 518.788154ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................................................XX..] duration: 469.872097ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:09:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 469.910216ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 529.047931ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:09:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 529.07191ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 402.775068ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:09:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 402.797109ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 576.789067ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:09:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 576.818225ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................................................XX.] duration: 460.848938ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:08:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 461.012519ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 504.785043ms trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:08:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 504.814693ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 31.988955186s trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:08:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 31.98897597s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 31.910200501s trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:08:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 31.910223831s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 544.200424ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:08:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 544.227602ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 15.833797211s trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:08:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 15.833833629s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.028682484s trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:08:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.028707447s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.023094929s trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:08:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.023125589s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.025711193s trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:08:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.025721958s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................................................XX] duration: 5.001543634s trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:08:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001584514s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.000631868s trace: https://jaeger.dev.storj.io/trace/3b9412a0c23f7ce8 FAIL (2025-11-28 22:07:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000658158s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [X..............] duration: 69.029305ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 28 Nov 2025 08:49:05 GMT Content-Type=text/html Content-Length=175 Set-Cookie=_cfuvid=D5QGD9NZ9gRZIBp19ZyTjMfXmtf7a.o3msoo8bUK2SQ-1764319745309-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare Cf-Ray=9a589c281a720c23-EWR Vary=Accept-Encoding Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/4b6d9636ae3cdbc FAIL (2025-11-28 08:59:30 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 69.051883ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 77.333409ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 28 Nov 2025 08:49:05 GMT Content-Type=text/html Content-Length=175 Set-Cookie=_cfuvid=D5QGD9NZ9gRZIBp19ZyTjMfXmtf7a.o3msoo8bUK2SQ-1764319745309-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare Cf-Ray=9a589c281a720c23-EWR Vary=Accept-Encoding Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/4b6d9636ae3cdbc FAIL (2025-11-28 08:58:58 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 77.359603ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..X............] duration: 80.063704ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 28 Nov 2025 08:49:05 GMT Content-Type=text/html Content-Length=175 Set-Cookie=_cfuvid=D5QGD9NZ9gRZIBp19ZyTjMfXmtf7a.o3msoo8bUK2SQ-1764319745309-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare Cf-Ray=9a589c281a720c23-EWR Vary=Accept-Encoding Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/4b6d9636ae3cdbc FAIL (2025-11-28 08:58:30 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 80.148385ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [...X...........] duration: 71.421415ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 28 Nov 2025 08:49:05 GMT Content-Type=text/html Content-Length=175 Set-Cookie=_cfuvid=D5QGD9NZ9gRZIBp19ZyTjMfXmtf7a.o3msoo8bUK2SQ-1764319745309-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare Cf-Ray=9a589c281a720c23-EWR Vary=Accept-Encoding Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/4b6d9636ae3cdbc FAIL (2025-11-28 08:57:24 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 71.445611ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [....X..........] duration: 72.01517ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 28 Nov 2025 08:49:05 GMT Content-Type=text/html Content-Length=175 Set-Cookie=_cfuvid=D5QGD9NZ9gRZIBp19ZyTjMfXmtf7a.o3msoo8bUK2SQ-1764319745309-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare Cf-Ray=9a589c281a720c23-EWR Vary=Accept-Encoding Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/4b6d9636ae3cdbc FAIL (2025-11-28 08:56:51 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 72.042728ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.....X.........] duration: 70.649366ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 28 Nov 2025 08:49:05 GMT Content-Type=text/html Content-Length=175 Set-Cookie=_cfuvid=D5QGD9NZ9gRZIBp19ZyTjMfXmtf7a.o3msoo8bUK2SQ-1764319745309-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare Cf-Ray=9a589c281a720c23-EWR Vary=Accept-Encoding Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/4b6d9636ae3cdbc FAIL (2025-11-28 08:56:24 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 70.668266ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 72.437413ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 28 Nov 2025 08:49:05 GMT Content-Type=text/html Content-Length=175 Set-Cookie=_cfuvid=D5QGD9NZ9gRZIBp19ZyTjMfXmtf7a.o3msoo8bUK2SQ-1764319745309-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare Cf-Ray=9a589c281a720c23-EWR Vary=Accept-Encoding Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/4b6d9636ae3cdbc FAIL (2025-11-28 08:55:37 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 72.460042ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 76.157152ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 28 Nov 2025 08:49:05 GMT Content-Type=text/html Content-Length=175 Set-Cookie=_cfuvid=D5QGD9NZ9gRZIBp19ZyTjMfXmtf7a.o3msoo8bUK2SQ-1764319745309-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare Cf-Ray=9a589c281a720c23-EWR Vary=Accept-Encoding Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/4b6d9636ae3cdbc FAIL (2025-11-28 08:55:21 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 76.180243ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 68.048021ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 28 Nov 2025 08:49:05 GMT Content-Type=text/html Content-Length=175 Set-Cookie=_cfuvid=D5QGD9NZ9gRZIBp19ZyTjMfXmtf7a.o3msoo8bUK2SQ-1764319745309-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare Cf-Ray=9a589c281a720c23-EWR Vary=Accept-Encoding Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/4b6d9636ae3cdbc FAIL (2025-11-28 08:54:17 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 68.168261ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 89.587539ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Fri, 28 Nov 2025 08:49:05 GMT Content-Type=text/html Content-Length=175 Set-Cookie=_cfuvid=D5QGD9NZ9gRZIBp19ZyTjMfXmtf7a.o3msoo8bUK2SQ-1764319745309-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare Cf-Ray=9a589c281a720c23-EWR Vary=Accept-Encoding Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/4b6d9636ae3cdbc FAIL (2025-11-28 08:53:10 UTC) orbiter warning: unexpected status code "500 Internal Server Error" "<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body>\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" Duration: 89.612308ms core website

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)