Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................................XX.............] duration:
455.672146ms trace:
https://jaeger.dev.storj.io/trace/1aeb398323fef20d
FAIL (2025-12-24 00:05:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 455.70294ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
533.449834ms trace:
https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0
FAIL (2025-12-24 00:05:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 533.48287ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
534.395301ms trace:
https://jaeger.dev.storj.io/trace/1c1b44184722d5aa
FAIL (2025-12-24 00:05:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload
Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload
Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments...
Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false
Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment...
Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment.
Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment
Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments...
Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments...
Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 534.42171ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
791.205112ms trace:
https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0
FAIL (2025-12-24 00:04:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 791.230551ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................................XX............] duration:
470.367582ms trace:
https://jaeger.dev.storj.io/trace/1aeb398323fef20d
FAIL (2025-12-24 00:04:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 470.395668ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
459.508998ms trace:
https://jaeger.dev.storj.io/trace/1c1b44184722d5aa
FAIL (2025-12-24 00:04:21 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload
Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload
Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments...
Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false
Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment...
Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment.
Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment
Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments...
Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments...
Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 459.593975ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................................XX...........] duration:
589.778145ms trace:
https://jaeger.dev.storj.io/trace/1aeb398323fef20d
FAIL (2025-12-24 00:03:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 589.801337ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
548.25419ms trace:
https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0
FAIL (2025-12-24 00:03:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 548.281854ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
500.057356ms trace:
https://jaeger.dev.storj.io/trace/1c1b44184722d5aa
FAIL (2025-12-24 00:03:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload
Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload
Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments...
Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false
Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment...
Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment.
Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment
Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments...
Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments...
Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 500.088079ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
479.142498ms trace:
https://jaeger.dev.storj.io/trace/1c1b44184722d5aa
FAIL (2025-12-24 00:03:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload
Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload
Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments...
Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false
Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment...
Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment.
Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment
Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments...
Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments...
Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 479.183709ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................................XX..........] duration:
472.603328ms trace:
https://jaeger.dev.storj.io/trace/1aeb398323fef20d
FAIL (2025-12-24 00:03:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 472.628301ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
581.105369ms trace:
https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0
FAIL (2025-12-24 00:02:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 581.124943ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
929.889175ms trace:
https://jaeger.dev.storj.io/trace/1c1b44184722d5aa
FAIL (2025-12-24 00:02:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload
Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload
Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments...
Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false
Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment...
Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment.
Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment
Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments...
Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments...
Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 929.915963ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................................XX.........] duration:
488.675079ms trace:
https://jaeger.dev.storj.io/trace/1aeb398323fef20d
FAIL (2025-12-24 00:01:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 488.790026ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
646.168115ms trace:
https://jaeger.dev.storj.io/trace/1c1b44184722d5aa
FAIL (2025-12-24 00:01:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload
Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload
Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments...
Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false
Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment...
Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment.
Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment
Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments...
Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments...
Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 646.197595ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
518.099283ms trace:
https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0
FAIL (2025-12-24 00:01:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 518.128067ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
424.785293ms trace:
https://jaeger.dev.storj.io/trace/1c1b44184722d5aa
FAIL (2025-12-24 00:01:13 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload
Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload
Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments...
Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false
Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment...
Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment.
Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment
Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments...
Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments...
Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 424.812651ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................................XX........] duration:
539.057746ms trace:
https://jaeger.dev.storj.io/trace/1aeb398323fef20d
FAIL (2025-12-24 00:00:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 539.083683ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
1.190289123s trace:
https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0
FAIL (2025-12-24 00:00:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.190310011s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
748.488772ms trace:
https://jaeger.dev.storj.io/trace/1c1b44184722d5aa
FAIL (2025-12-24 00:00:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload
Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload
Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments...
Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false
Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment...
Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment.
Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment
Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments...
Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments...
Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 748.519581ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
1.071325025s trace:
https://jaeger.dev.storj.io/trace/1c1b44184722d5aa
FAIL (2025-12-23 23:59:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload
Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload
Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments...
Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false
Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment...
Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment.
Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment
Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments...
Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments...
Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1.071378832s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................................XX.......] duration:
481.049205ms trace:
https://jaeger.dev.storj.io/trace/1aeb398323fef20d
FAIL (2025-12-23 23:59:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 481.0664ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
545.372818ms trace:
https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0
FAIL (2025-12-23 23:59:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 545.400738ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
398.651338ms trace:
https://jaeger.dev.storj.io/trace/1c1b44184722d5aa
FAIL (2025-12-23 23:59:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload
Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload
Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments...
Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false
Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment...
Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment.
Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment
Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments...
Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments...
Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 398.6761ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................................XX......] duration:
504.641242ms trace:
https://jaeger.dev.storj.io/trace/1aeb398323fef20d
FAIL (2025-12-23 23:59:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 504.668594ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
925.993554ms trace:
https://jaeger.dev.storj.io/trace/1c1b44184722d5aa
FAIL (2025-12-23 23:58:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload
Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload
Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments...
Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false
Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment...
Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment.
Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment
Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments...
Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments...
Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 926.01737ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
669.273629ms trace:
https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0
FAIL (2025-12-23 23:58:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 669.38814ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration:
471.372909ms trace:
https://jaeger.dev.storj.io/trace/1aeb398323fef20d
FAIL (2025-12-23 23:58:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 471.401667ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................................XX....] duration:
631.851348ms trace:
https://jaeger.dev.storj.io/trace/1aeb398323fef20d
FAIL (2025-12-23 23:58:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 631.933235ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
1.115863552s trace:
https://jaeger.dev.storj.io/trace/1c1b44184722d5aa
FAIL (2025-12-23 23:57:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload
Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload
Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments...
Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false
Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment...
Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment.
Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment
Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments...
Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments...
Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1.115889919s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
534.502703ms trace:
https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0
FAIL (2025-12-23 23:57:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 534.698005ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
483.234606ms trace:
https://jaeger.dev.storj.io/trace/1c1b44184722d5aa
FAIL (2025-12-23 23:57:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload
Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload
Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments...
Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false
Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment...
Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment.
Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment
Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments...
Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments...
Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 483.26392ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................................................XX...] duration:
463.970055ms trace:
https://jaeger.dev.storj.io/trace/1aeb398323fef20d
FAIL (2025-12-23 23:57:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 464.014914ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
400.613689ms trace:
https://jaeger.dev.storj.io/trace/1c1b44184722d5aa
FAIL (2025-12-23 23:56:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload
Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload
Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments...
Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false
Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment...
Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment.
Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment
Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments...
Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments...
Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 400.671321ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
527.589493ms trace:
https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0
FAIL (2025-12-23 23:56:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 527.688615ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
526.11898ms trace:
https://jaeger.dev.storj.io/trace/1c1b44184722d5aa
FAIL (2025-12-23 23:56:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload
Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload
Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments...
Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false
Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment...
Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment.
Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment
Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments...
Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments...
Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 526.14977ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
828.085589ms trace:
https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0
FAIL (2025-12-23 23:56:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 828.124534ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................................................XX..] duration:
509.974434ms trace:
https://jaeger.dev.storj.io/trace/1aeb398323fef20d
FAIL (2025-12-23 23:56:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 510.000065ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
461.822316ms trace:
https://jaeger.dev.storj.io/trace/1c1b44184722d5aa
FAIL (2025-12-23 23:55:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload
Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload
Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments...
Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false
Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment...
Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment.
Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment
Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments...
Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments...
Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 461.907014ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
567.788606ms trace:
https://jaeger.dev.storj.io/trace/1c1b44184722d5aa
FAIL (2025-12-23 23:55:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload
Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload
Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments...
Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false
Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment...
Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment.
Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment
Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments...
Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments...
Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 567.810858ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................................................XX.] duration:
823.762699ms trace:
https://jaeger.dev.storj.io/trace/1aeb398323fef20d
FAIL (2025-12-23 23:55:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 823.780458ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
30.003504425s trace:
https://jaeger.dev.storj.io/trace/1aeb398323fef20d
FAIL (2025-12-23 23:55:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.003521623s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
719.916894ms trace:
https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0
FAIL (2025-12-23 23:54:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 719.944936ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.000643384s trace:
https://jaeger.dev.storj.io/trace/555c6326c66c5d6
FAIL (2025-12-23 23:54:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.00066935s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
640.676168ms trace:
https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0
FAIL (2025-12-23 23:53:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 640.699989ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.00188863s trace:
https://jaeger.dev.storj.io/trace/12c7b79ad5f0f9f0
FAIL (2025-12-23 23:53:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001916572s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
981.439015ms trace:
https://jaeger.dev.storj.io/trace/1c1b44184722d5aa
FAIL (2025-12-23 23:53:46 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload
Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload
Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments...
Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false
Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment...
Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment.
Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment
Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments...
Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments...
Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 981.454348ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.026796163s trace:
https://jaeger.dev.storj.io/trace/1c1b44184722d5aa
FAIL (2025-12-23 23:53:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Dec 23 23:52:45.354395589 (440h24m46.182103983s) [upload=458210]: Starting upload
Dec 23 23:52:45.354532150 (440h24m46.182240542s) [upload=458210]: Done starting upload
Dec 23 23:52:45.354594348 (440h24m46.18230274s) [upload=458210]: Uploading segments...
Dec 23 23:52:45.354916909 (440h24m46.1826253s) [upload=458210]: Got next segment. Inline: false
Dec 23 23:52:45.354962581 (440h24m46.182670962s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 23 23:52:45.354974323 (440h24m46.182682715s) [upload=458210]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 23 23:52:45.379894904 (440h24m46.207603304s) [seg_pos={0 0}, upload=458210]: Begin upload segment...
Dec 23 23:52:45.379990233 (440h24m46.207698632s) [seg_pos={0 0}, upload=458210]: Done begin upload segment.
Dec 23 23:52:45.380050641 (440h24m46.207759039s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.380172631 (440h24m46.207881034s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381580640 (440h24m46.209289042s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.381902585 (440h24m46.209610985s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.382642448 (440h24m46.210350851s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383306354 (440h24m46.211014754s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.383775650 (440h24m46.211484099s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384114460 (440h24m46.211822852s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384438670 (440h24m46.212147064s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384821354 (440h24m46.212529756s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.384934655 (440h24m46.212643056s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385176034 (440h24m46.212884415s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385536456 (440h24m46.213244838s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385564344 (440h24m46.213272742s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.385887274 (440h24m46.213595671s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386145335 (440h24m46.213853737s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386240101 (440h24m46.213948495s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.386697484 (440h24m46.214405886s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Uploading piece...
Dec 23 23:52:45.450218774 (440h24m46.277927167s) [upload=458210]: Next returned nil segment
Dec 23 23:52:45.450237550 (440h24m46.277945951s) [upload=458210]: Waiting for error group managing segments...
Dec 23 23:52:45.454985760 (440h24m46.282694157s) [noise=true , address=145.79.227.49:4349 , piece_id=7AHF2VARNFABN4TIXT6LSHSDFPBGSIPA7A7QSZFIO2J7KBBSMOVA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.455539664 (440h24m46.283248076s) [noise=true , address=77.90.31.19:4872 , piece_id=5SR5NWJ7PMZPP44KFKVQFE576KAXO5SRK2HMMUCXKZ72RWZQXJYA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.457117407 (440h24m46.284825801s) [noise=true , address=145.79.227.36:4527 , piece_id=653QQK7Z53VLWDUASWTCZ6WDEA5FI6RQGJL3FK7ZQZLZCCRGHIAQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.464356838 (440h24m46.292065237s) [noise=true , address=77.90.31.18:4813 , piece_id=PAIEFFS2LP4AOAGM24NGE4BJKEKBETBCR43TYYJLPX36COBXOB3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.469585832 (440h24m46.297294233s) [noise=true , address=77.90.31.21:4881 , piece_id=TDMO72IOLCLT3PJZ3DUWB2B4SISOVXJDCUKJZVZLE7WQ6XJ4TARQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.478043358 (440h24m46.305751758s) [noise=true , address=77.90.31.12:4603 , piece_id=J6OXFDSGYDT543Z2FOD7PA2IIYS3RUT5JJRLINCMRQCTJVJ5N2YA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.480811556 (440h24m46.308519937s) [noise=true , address=145.79.227.13:4831 , piece_id=6LGRNHO3D6HXICHSUUATVDXEU3YORPCAWBYHUYI5AMJGFPOIZMJQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.600999598 (440h24m46.428707989s) [noise=true , address=145.79.227.43:4160 , piece_id=VTYFFIAW6I43AQB7RV3GNYJYG6OD7JWH6C3S5LYJKP5BND77WBBQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:45.653470064 (440h24m46.481178456s) [noise=true , address=145.79.227.19:4089 , piece_id=L225EP4GOQT4QAMNRMTGQX74Z5HJGZR353W6OVV7NE5K36PAXI6Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.379483343 (440h24m47.20719174s) [noise=true , address=145.79.227.30:4789 , piece_id=WQ6A2AQLDUHGANDGW65Y3TNBPN5SA3NMZL3AYEP3UECO74U6YHOQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:46.516369920 (440h24m47.344078313s) [noise=true , address=145.79.227.44:4202 , piece_id=HXVNPBTB7ISE2QBUZ2FTWQ32IZWXCDA4LLGZAESR6RCYFROBQ4UQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:52:51.143730771 (440h24m51.971439151s) [noise=true , address=64.49.13.169:4709 , piece_id=IHSKVS644YH4B6VDRQZEYJGPJTHP2CDX3WHMTQVO6YN7DOEQT4JA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:38.021537112 (440h25m38.849245507s) [noise=true , address=45.91.116.8:4059 , piece_id=BYMYDEG6JP6PTXEQUDAM5J6YWRTQM6LHWKOPDNZ4YOP5ZWAGJI5A, seg_pos={0 0}, upload=458210]: Done uploading piece. err: <nil>
Dec 23 23:53:45.313877820 (440h25m46.141586214s) [noise=true , address=45.91.116.9:4283 , piece_id=3TRA44ZW6XRAS4AUW7QHIL6CAN4H4TZZGAGUZHIBHHATMPDPJXPQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g, address:45.91.116.9:4283): protocol: write tcp 159.223.143.39:33124->45.91.116.9:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.313897907 (440h25m46.141606288s) [noise=true , address=45.91.116.5:4325 , piece_id=KZXUA7DYJ4YQMREMH577IHGZC5FP6NFDEI2A6GNTZ4G25IN34AAA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1ZLq4LtsWRzLuBfckTemv9ZSzPnNo37tQYtfmXj2bntQAuuuaC, address:45.91.116.5:4325): protocol: write tcp 159.223.143.39:47700->45.91.116.5:4325: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314066057 (440h25m46.141774458s) [noise=true , address=45.91.116.12:4250 , piece_id=I274CBRQL3V7W7RF65NE4XN66KRLOIGDMP2VVYYV2ENSHVNFFYPA, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1KvoMM3ptQXSxPhXFe3UoS145h8n3DMT9qmeaoR3bqNa8KLMm9, address:45.91.116.12:4250): protocol: write tcp 159.223.143.39:40436->45.91.116.12:4250: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314195784 (440h25m46.141904181s) [noise=true , address=45.91.116.6:4987 , piece_id=4FFKOVOCLH7EIPHOGL5MDYZFJJNLOLWFBVD2XVTT42YE3AHTQU3Q, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: rpc: dial tcp 45.91.116.6:4987: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314448583 (440h25m46.142156983s) [noise=true , address=45.91.116.11:4138 , piece_id=GQTSUONBLMRVI2NKKTHOC236LW4D5Z3RUSMMU7FBIZRXNKKY5WRQ, seg_pos={0 0}, upload=458210]: Done uploading piece. err: ecclient: upload failed (node:1rQNEmGLtqhxcDAeFocRDsfkr5oZMkbLVhmumwb5BKkreNiDPU, address:45.91.116.11:4138): protocol: rpc: dial tcp 45.91.116.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 23:53:45.314489158 (440h25m46.142197549s) [seg_pos={0 0}, upload=458210]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 23 23:53:45.338100126 (440h25m46.165808516s) [upload=458210]: Done uploading segments...
Dec 23 23:53:45.338144200 (440h25m46.165852594s) [upload=458210]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1m0.026820133s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [X...........................................................] duration:
278.600978ms remote:
"tcp:136.0.77.4:443" target:
https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1b5ea30191a6df89
FAIL (2025-12-23 23:40:51 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 278.631908ms
edge
linksharing
bgp
ap1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.ap1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [.X..........................................................] duration:
278.748319ms remote:
"tcp:136.0.77.4:443" target:
https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1b5ea30191a6df89
FAIL (2025-12-23 23:39:46 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwlk53gfhsby563unjoqoivp56jq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 278.785394ms
edge
linksharing
bgp
ap1
|