Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................XX..............................] duration: 464.413183ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:17:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 464.457849ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 582.516785ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:17:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 582.531873ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 537.109965ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:17:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 537.138371ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................XX.............................] duration: 456.598283ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:17:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 456.624514ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 522.617019ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:17:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 522.646493ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 414.110829ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:16:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 414.139141ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 531.563816ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:16:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 531.682701ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................XX............................] duration: 465.289302ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:15:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 465.332624ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 886.699006ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:15:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 886.72713ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 556.936376ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:15:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 557.031539ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 480.626168ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:15:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 480.655274ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................XX...........................] duration: 452.881809ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:15:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 453.393806ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 573.3182ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:15:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 573.345745ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 569.11381ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:14:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 569.143167ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................XX..........................] duration: 484.940917ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:14:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 484.975449ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 425.871226ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:14:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 425.907658ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 538.069853ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:13:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 538.112393ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................XX.........................] duration: 477.820232ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:13:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 477.849829ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 415.241367ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:13:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 415.268813ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 621.530282ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:13:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 621.55808ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................XX........................] duration: 460.555381ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:12:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 460.577787ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 396.472131ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:12:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 396.507911ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 895.537833ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:12:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 895.565188ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 602.444296ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:11:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 602.589955ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................XX.......................] duration: 456.531726ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:11:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 456.56741ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.038436999s trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:11:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.038472487s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................XX......................] duration: 441.661579ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:11:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 441.688469ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 974.240633ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:11:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 974.272242ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 645.188129ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:10:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 645.223793ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 414.457707ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:10:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 414.487899ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................XX.....................] duration: 487.96388ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:10:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 487.988886ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 556.987015ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:10:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 557.018625ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 851.389165ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:09:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 851.413319ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................XX....................] duration: 477.987296ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:09:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 478.026291ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 570.792755ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:09:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 570.85995ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................XX...................] duration: 486.026556ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:08:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 486.062869ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 465.694431ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:08:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 465.730245ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................XX..................] duration: 450.891778ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:08:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 450.946336ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 522.795329ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:08:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 522.856621ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................XX.................] duration: 510.264189ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:07:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 510.290202ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 415.239984ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:07:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 415.272803ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 546.426804ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:07:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 546.452948ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 626.142148ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:07:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 626.156002ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................XX................] duration: 488.924618ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:07:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 488.968941ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 397.784435ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:06:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 397.813793ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 560.963676ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:06:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 560.996407ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................XX...............] duration: 459.376339ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:06:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 459.407414ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................XX..............] duration: 508.703057ms trace: https://jaeger.dev.storj.io/trace/1aeb398323fef20d FAIL (2025-12-24 00:06:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 508.798054ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 361.069174ms trace: https://jaeger.dev.storj.io/trace/1c1b44184722d5aa FAIL (2025-12-24 00:06:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments... Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment... Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment. Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece... Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments... Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil> Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments... Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 361.095464ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 610.830749ms trace: https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0 FAIL (2025-12-24 00:06:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 610.859429ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)