Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 471.942548ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:30:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 471.967089ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 916.74459ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:30:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 916.781235ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 471.86498ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:29:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 471.87923ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 552.861854ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:29:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 552.891821ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 598.210844ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:29:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 598.334729ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 514.909786ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:29:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 514.934228ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............XX...........................................] duration: 444.253389ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:28:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 444.284604ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 534.913496ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:28:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 534.942698ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 581.677183ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:28:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 581.703769ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 532.354284ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:28:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 532.388602ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................XX..........................................] duration: 451.537562ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:27:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 451.577822ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 883.486042ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:27:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 883.511146ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................XX.........................................] duration: 445.391803ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:27:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 445.413752ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 539.401581ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:27:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 539.433374ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 504.476414ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:26:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 504.547166ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 565.869232ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:26:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 565.938791ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 886.50011ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:26:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 886.538639ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................XX........................................] duration: 484.98801ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:26:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 485.018243ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 400.462264ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:25:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 400.489032ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 921.781439ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:25:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 921.808689ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................XX.......................................] duration: 435.681196ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:25:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 435.783925ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 591.158135ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:25:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 591.182382ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 972.245443ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:24:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 972.321775ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 443.808891ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:24:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 443.837787ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................XX......................................] duration: 471.545536ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:24:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 471.587971ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 1.073690319s trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:23:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.073719978s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 406.869288ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:23:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 406.901506ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration: 432.851654ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:23:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 432.876515ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 453.880591ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:22:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 453.906253ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 470.101166ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:22:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 470.145131ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 553.486132ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:22:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 553.51369ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................XX....................................] duration: 483.131297ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:22:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 483.162401ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 570.949374ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:22:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 570.980948ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 746.18268ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:22:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 746.211133ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................XX...................................] duration: 486.998588ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:21:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 487.025574ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 585.503016ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:21:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 585.521011ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 717.657826ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:21:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 717.689874ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................XX..................................] duration: 448.079667ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:21:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 448.122689ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 955.243692ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:21:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 955.347949ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 557.356196ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:20:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 557.959745ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 655.844041ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:20:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 655.874715ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................XX.................................] duration: 473.342045ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:20:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 473.841572ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 495.65729ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:19:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 495.682437ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 1.232004764s trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:19:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.232041461s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................XX................................] duration: 463.224668ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:19:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 463.321316ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 611.91221ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:19:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 611.946392ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................XX...............................] duration: 446.461114ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:18:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 446.487855ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 505.347952ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:18:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 505.372104ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 471.006691ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:18:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 471.019796ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 593.617549ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:18:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 593.634966ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)