Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 442.958542ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:24:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 442.982405ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 357.29655ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:23:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 357.320434ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................................XX.....................] duration: 593.569727ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:23:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 593.592091ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 470.339745ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:23:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 470.362023ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 563.056291ms trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:23:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 563.085246ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 518.75177ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:22:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 518.778146ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 402.876377ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:22:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 402.904057ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 408.943467ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:22:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 408.97226ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................................XX....................] duration: 513.262204ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:22:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 513.314ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 567.32935ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:22:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 567.36579ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 487.215632ms trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:21:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 487.24448ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 361.34668ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:21:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 361.369918ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 473.713557ms trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:21:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 473.735019ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 516.397815ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:21:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 516.424542ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................................XX...................] duration: 408.264358ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:21:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 408.287762ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 540.613134ms trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:20:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 540.632967ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 405.70553ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:20:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 405.743556ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 523.322233ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:20:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 523.346355ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................................XX..................] duration: 422.789571ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:20:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 422.810091ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................................XX.................] duration: 495.127154ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:20:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 495.174032ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 427.385704ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:19:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 427.410984ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 475.452419ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:19:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 475.510389ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................................XX................] duration: 670.89912ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:19:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 670.939612ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 497.161098ms trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:19:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 497.187969ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................................XX...............] duration: 437.131108ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:18:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 437.155183ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 495.526796ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:18:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 495.5629ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 380.860092ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:18:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 380.886827ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 515.117815ms trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:18:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 515.143303ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................................XX..............] duration: 420.566887ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:18:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 420.61332ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 543.064557ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:18:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 543.083598ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................................XX.............] duration: 487.128096ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:17:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 487.146866ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 512.685421ms trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:17:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 512.710652ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 363.260698ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:17:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 363.306907ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................................XX............] duration: 458.313141ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:17:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 458.341421ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 506.116206ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:17:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 506.136878ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 335.760183ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:17:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 335.790904ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 509.298197ms trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:17:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 509.324727ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................................XX...........] duration: 442.268527ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:16:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 442.302336ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 377.21821ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:16:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 377.243206ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 493.996101ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:16:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 494.026203ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 492.996136ms trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:15:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 493.022933ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................................XX..........] duration: 435.25747ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:15:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 435.348413ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 512.029341ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:15:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 512.054227ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 387.205034ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:15:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 387.220752ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................................XX.........] duration: 455.616702ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:15:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 455.638766ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 424.405968ms trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:14:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 424.432552ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 547.348346ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:14:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 547.372198ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................................XX........] duration: 415.961011ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:14:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 415.981333ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 573.537077ms trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:14:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 573.571097ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 414.26727ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:14:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 414.292997ms core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)