Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.X...X.............X............X.......X...................] duration:
478.898936ms trace:
https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91
FAIL (2025-10-31 21:37:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 478.927292ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [.X...................................X.......X..............] duration:
488.28379ms trace:
https://jaeger.dev.storj.io/trace/689f32b5a76d000d
FAIL (2025-10-31 21:36:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 488.40505ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..........................X..........X......................] duration:
584.205524ms trace:
https://jaeger.dev.storj.io/trace/464ff9dd561aefd2
FAIL (2025-10-31 21:36:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 584.226065ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................................X.....X.............] duration:
412.72942ms trace:
https://jaeger.dev.storj.io/trace/22de3cd654292a7d
FAIL (2025-10-31 21:36:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload
Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload
Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments...
Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false
Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment...
Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment.
Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment
Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments...
Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments...
Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 412.756875ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [..X...................................X.......X.............] duration:
488.11111ms trace:
https://jaeger.dev.storj.io/trace/689f32b5a76d000d
FAIL (2025-10-31 21:36:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 488.135053ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..X...X.............X............X.......X..................] duration:
551.596298ms trace:
https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91
FAIL (2025-10-31 21:36:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 551.630957ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [............X.............X.....................X...........] duration:
421.738408ms trace:
https://jaeger.dev.storj.io/trace/2a65f5f21f735664
FAIL (2025-10-31 21:36:21 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload
Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload
Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments...
Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false
Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment...
Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment.
Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment
Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments...
Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments...
Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 421.763135ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................................X.....X............] duration:
432.621569ms trace:
https://jaeger.dev.storj.io/trace/22de3cd654292a7d
FAIL (2025-10-31 21:36:12 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload
Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload
Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments...
Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false
Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment...
Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment.
Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment
Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments...
Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments...
Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 432.641818ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................X.................X..........] duration:
500.828552ms trace:
https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55
FAIL (2025-10-31 21:36:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded
======LOGS======
825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment
Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments...
Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection
Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments...
Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil>
Duration: 500.946599ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.............X.............X.....................X..........] duration:
465.594988ms trace:
https://jaeger.dev.storj.io/trace/2a65f5f21f735664
FAIL (2025-10-31 21:35:57 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload
Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload
Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments...
Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false
Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment...
Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment.
Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment
Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments...
Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments...
Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 465.619261ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [...X...................................X.......X............] duration:
750.515147ms trace:
https://jaeger.dev.storj.io/trace/689f32b5a76d000d
FAIL (2025-10-31 21:35:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 750.537243ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...........................X..........X.....................] duration:
607.130225ms trace:
https://jaeger.dev.storj.io/trace/464ff9dd561aefd2
FAIL (2025-10-31 21:35:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 607.154684ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...X...X.............X............X.......X.................] duration:
480.059565ms trace:
https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91
FAIL (2025-10-31 21:35:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 480.084428ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................X.....X...........] duration:
443.601522ms trace:
https://jaeger.dev.storj.io/trace/22de3cd654292a7d
FAIL (2025-10-31 21:35:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload
Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload
Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments...
Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false
Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment...
Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment.
Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment
Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments...
Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments...
Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 443.642016ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................X.................X.........] duration:
471.945108ms trace:
https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55
FAIL (2025-10-31 21:35:23 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded
======LOGS======
825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment
Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments...
Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection
Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments...
Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil>
Duration: 471.970619ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [............................X..........X....................] duration:
604.028512ms trace:
https://jaeger.dev.storj.io/trace/464ff9dd561aefd2
FAIL (2025-10-31 21:35:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 604.896551ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.............................X..........X...................] duration:
665.15429ms trace:
https://jaeger.dev.storj.io/trace/464ff9dd561aefd2
FAIL (2025-10-31 21:35:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 665.524681ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [....X...X.............X............X.......X................] duration:
449.845877ms trace:
https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91
FAIL (2025-10-31 21:35:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 449.873612ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [....X...................................X.......X...........] duration:
528.605931ms trace:
https://jaeger.dev.storj.io/trace/689f32b5a76d000d
FAIL (2025-10-31 21:35:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 528.635514ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..............X.............X.....................X.........] duration:
420.166067ms trace:
https://jaeger.dev.storj.io/trace/2a65f5f21f735664
FAIL (2025-10-31 21:34:47 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload
Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload
Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments...
Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false
Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment...
Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment.
Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment
Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments...
Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments...
Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 420.196246ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................X.................X........] duration:
430.461275ms trace:
https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55
FAIL (2025-10-31 21:34:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded
======LOGS======
825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment
Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments...
Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection
Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments...
Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil>
Duration: 430.643712ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..............................X..........X..................] duration:
564.555088ms trace:
https://jaeger.dev.storj.io/trace/464ff9dd561aefd2
FAIL (2025-10-31 21:34:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 564.597883ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................................X.....X..........] duration:
400.451628ms trace:
https://jaeger.dev.storj.io/trace/22de3cd654292a7d
FAIL (2025-10-31 21:34:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload
Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload
Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments...
Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false
Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment...
Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment.
Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment
Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments...
Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments...
Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 400.480414ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....X...X.............X............X.......X...............] duration:
448.002532ms trace:
https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91
FAIL (2025-10-31 21:34:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 448.030737ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...............X.............X.....................X........] duration:
696.320503ms trace:
https://jaeger.dev.storj.io/trace/2a65f5f21f735664
FAIL (2025-10-31 21:34:00 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload
Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload
Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments...
Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false
Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment...
Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment.
Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment
Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments...
Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments...
Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 696.543206ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [X....X...................................X.......X..........] duration:
545.909003ms trace:
https://jaeger.dev.storj.io/trace/689f32b5a76d000d
FAIL (2025-10-31 21:33:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 545.931095ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...............................X..........X.................] duration:
640.789401ms trace:
https://jaeger.dev.storj.io/trace/464ff9dd561aefd2
FAIL (2025-10-31 21:33:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 640.817459ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................X.................X.......] duration:
493.651124ms trace:
https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55
FAIL (2025-10-31 21:33:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded
======LOGS======
825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment
Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments...
Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection
Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments...
Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil>
Duration: 493.683647ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................................X.....X.........] duration:
404.288338ms trace:
https://jaeger.dev.storj.io/trace/22de3cd654292a7d
FAIL (2025-10-31 21:33:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload
Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload
Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments...
Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false
Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment...
Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment.
Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment
Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments...
Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments...
Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 404.31481ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.X....X...................................X.......X.........] duration:
485.139969ms trace:
https://jaeger.dev.storj.io/trace/689f32b5a76d000d
FAIL (2025-10-31 21:33:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 485.167092ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [................................X..........X................] duration:
777.485435ms trace:
https://jaeger.dev.storj.io/trace/464ff9dd561aefd2
FAIL (2025-10-31 21:33:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 777.662036ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [......X...X.............X............X.......X..............] duration:
487.602528ms trace:
https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91
FAIL (2025-10-31 21:33:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 487.628457ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [................X.............X.....................X.......] duration:
434.792658ms trace:
https://jaeger.dev.storj.io/trace/2a65f5f21f735664
FAIL (2025-10-31 21:33:08 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload
Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload
Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments...
Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false
Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment...
Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment.
Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment
Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments...
Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments...
Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 434.843773ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................X.................X......] duration:
612.879706ms trace:
https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55
FAIL (2025-10-31 21:32:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded
======LOGS======
825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment
Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments...
Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection
Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments...
Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil>
Duration: 612.907138ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................................X.....X........] duration:
393.249682ms trace:
https://jaeger.dev.storj.io/trace/22de3cd654292a7d
FAIL (2025-10-31 21:32:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload
Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload
Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments...
Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false
Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment...
Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment.
Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment
Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments...
Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments...
Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 393.269859ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.................................X..........X...............] duration:
617.99019ms trace:
https://jaeger.dev.storj.io/trace/464ff9dd561aefd2
FAIL (2025-10-31 21:32:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 618.019571ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..X....X...................................X.......X........] duration:
487.378415ms trace:
https://jaeger.dev.storj.io/trace/689f32b5a76d000d
FAIL (2025-10-31 21:32:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 487.400744ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.................X.............X.....................X......] duration:
422.610789ms trace:
https://jaeger.dev.storj.io/trace/2a65f5f21f735664
FAIL (2025-10-31 21:32:19 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload
Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload
Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments...
Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false
Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment...
Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment.
Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment
Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments...
Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments...
Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 422.63817ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......X...X.............X............X.......X.............] duration:
436.702053ms trace:
https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91
FAIL (2025-10-31 21:32:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 436.725242ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................................X.....X.......] duration:
383.461872ms trace:
https://jaeger.dev.storj.io/trace/22de3cd654292a7d
FAIL (2025-10-31 21:32:13 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload
Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload
Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments...
Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false
Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment...
Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment.
Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment
Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments...
Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments...
Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 383.485476ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................X.................X.....] duration:
422.389461ms trace:
https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55
FAIL (2025-10-31 21:32:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded
======LOGS======
825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment
Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments...
Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection
Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments...
Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil>
Duration: 422.415075ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [........X...X.............X............X.......X............] duration:
438.020794ms trace:
https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91
FAIL (2025-10-31 21:31:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 438.046793ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..................................X..........X..............] duration:
612.931109ms trace:
https://jaeger.dev.storj.io/trace/464ff9dd561aefd2
FAIL (2025-10-31 21:31:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 612.961139ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...X....X...................................X.......X.......] duration:
503.332794ms trace:
https://jaeger.dev.storj.io/trace/689f32b5a76d000d
FAIL (2025-10-31 21:31:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 505.799727ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..................X.............X.....................X.....] duration:
430.795935ms trace:
https://jaeger.dev.storj.io/trace/2a65f5f21f735664
FAIL (2025-10-31 21:31:27 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload
Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload
Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments...
Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false
Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment...
Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment.
Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment
Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments...
Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments...
Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 430.820665ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...................................X..........X.............] duration:
621.475842ms trace:
https://jaeger.dev.storj.io/trace/464ff9dd561aefd2
FAIL (2025-10-31 21:31:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 621.529017ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................................X.....X......] duration:
406.853304ms trace:
https://jaeger.dev.storj.io/trace/22de3cd654292a7d
FAIL (2025-10-31 21:31:11 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload
Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload
Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments...
Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false
Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment...
Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment.
Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment
Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments...
Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments...
Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 406.879348ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................X.................X....] duration:
503.330295ms trace:
https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55
FAIL (2025-10-31 21:31:04 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded
======LOGS======
825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment
Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments...
Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection
Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments...
Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil>
Duration: 503.371326ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [....X....X...................................X.......X......] duration:
482.108396ms trace:
https://jaeger.dev.storj.io/trace/689f32b5a76d000d
FAIL (2025-10-31 21:30:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 482.131645ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.........X...X.............X............X.......X...........] duration:
479.066342ms trace:
https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91
FAIL (2025-10-31 21:30:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 479.106999ms
edge
gateway
us-select-1
|