Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 148722 reports (Page 72 of 2975)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.....X.................................] duration: 25.825023578s trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 21:54:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 25.825047321s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.....................X.................................X.] duration: 573.549114ms trace: https://jaeger.dev.storj.io/trace/236699c64467bb84 FAIL (2025-10-31 21:54:50 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:54.316715391 (34h14m24.980664371s) [upload=35593]: Starting upload Oct 31 21:53:54.316916021 (34h14m24.980865001s) [upload=35593]: Done starting upload Oct 31 21:53:54.316999602 (34h14m24.980948574s) [upload=35593]: Uploading segments... Oct 31 21:53:54.317521796 (34h14m24.981470776s) [upload=35593]: Got next segment. Inline: false Oct 31 21:53:54.317593157 (34h14m24.98154214s) [upload=35593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:54.317600463 (34h14m24.981549444s) [upload=35593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:54.344972306 (34h14m25.008921293s) [seg_pos={0 0}, upload=35593]: Begin upload segment... Oct 31 21:53:54.345030267 (34h14m25.008979247s) [seg_pos={0 0}, upload=35593]: Done begin upload segment. Oct 31 21:53:54.345213856 (34h14m25.009162844s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345225337 (34h14m25.009174327s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345830005 (34h14m25.009778994s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346300953 (34h14m25.010249931s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346533214 (34h14m25.010482193s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347118612 (34h14m25.011067594s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347365203 (34h14m25.011314187s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347568970 (34h14m25.01151796s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347824532 (34h14m25.01177352s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347959678 (34h14m25.011908664s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348214346 (34h14m25.012163334s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348367787 (34h14m25.012316775s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348778049 (34h14m25.012727028s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349186865 (34h14m25.013135844s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349752289 (34h14m25.013701272s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.352239055 (34h14m25.016188036s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.419826216 (34h14m25.083775196s) [upload=35593]: Next returned nil segment Oct 31 21:53:54.419839894 (34h14m25.083788873s) [upload=35593]: Waiting for error group managing segments... Oct 31 21:53:54.425324284 (34h14m25.089273265s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426679856 (34h14m25.090628835s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426719775 (34h14m25.090668754s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427269718 (34h14m25.091218697s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427363119 (34h14m25.0913121s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427761264 (34h14m25.091710245s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428214622 (34h14m25.092163602s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428234634 (34h14m25.092183614s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428729183 (34h14m25.092678166s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428832310 (34h14m25.092781292s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.429592128 (34h14m25.093541127s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.447797690 (34h14m25.111746672s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:54:24.282494056 (34h14m54.946443036s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: read tcp 159.223.143.39:35840->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282509279 (34h14m54.946458269s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 159.223.143.39:36998->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282717367 (34h14m54.946666355s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 159.223.143.39:55516->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282833181 (34h14m54.946782153s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 159.223.143.39:35952->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282879533 (34h14m54.946828499s) [seg_pos={0 0}, upload=35593]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:24.306226453 (34h14m54.970175433s) [upload=35593]: Done uploading segments... Oct 31 21:54:24.306271889 (34h14m54.970220869s) [upload=35593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 573.577727ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X.......X......................................X] duration: 30.000925048s trace: https://jaeger.dev.storj.io/trace/3adce5f38013470e FAIL (2025-10-31 21:54:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000946659s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.....................X.................................X] duration: 30.026333564s trace: https://jaeger.dev.storj.io/trace/236699c64467bb84 FAIL (2025-10-31 21:54:24 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:53:54.316715391 (34h14m24.980664371s) [upload=35593]: Starting upload Oct 31 21:53:54.316916021 (34h14m24.980865001s) [upload=35593]: Done starting upload Oct 31 21:53:54.316999602 (34h14m24.980948574s) [upload=35593]: Uploading segments... Oct 31 21:53:54.317521796 (34h14m24.981470776s) [upload=35593]: Got next segment. Inline: false Oct 31 21:53:54.317593157 (34h14m24.98154214s) [upload=35593]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:53:54.317600463 (34h14m24.981549444s) [upload=35593]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:53:54.344972306 (34h14m25.008921293s) [seg_pos={0 0}, upload=35593]: Begin upload segment... Oct 31 21:53:54.345030267 (34h14m25.008979247s) [seg_pos={0 0}, upload=35593]: Done begin upload segment. Oct 31 21:53:54.345213856 (34h14m25.009162844s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345225337 (34h14m25.009174327s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.345830005 (34h14m25.009778994s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346300953 (34h14m25.010249931s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.346533214 (34h14m25.010482193s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347118612 (34h14m25.011067594s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347365203 (34h14m25.011314187s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347568970 (34h14m25.01151796s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347824532 (34h14m25.01177352s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.347959678 (34h14m25.011908664s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348214346 (34h14m25.012163334s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348367787 (34h14m25.012316775s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.348778049 (34h14m25.012727028s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349186865 (34h14m25.013135844s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.349752289 (34h14m25.013701272s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.352239055 (34h14m25.016188036s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Uploading piece... Oct 31 21:53:54.419826216 (34h14m25.083775196s) [upload=35593]: Next returned nil segment Oct 31 21:53:54.419839894 (34h14m25.083788873s) [upload=35593]: Waiting for error group managing segments... Oct 31 21:53:54.425324284 (34h14m25.089273265s) [noise=true , address=145.79.227.21:4474 , piece_id=SL3ZMJFSXOWA4SI24IXL3LKAN5W2GPQT7RHWN24MUZ74HWPH24DA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426679856 (34h14m25.090628835s) [noise=true , address=145.79.227.25:4618 , piece_id=PYMAXYCOBDSEPJHEYFVJRWRWFY4CD3WLULB2DR6J6FYUPXGPK3IA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.426719775 (34h14m25.090668754s) [noise=true , address=145.79.227.13:4801 , piece_id=WA4UDQOPK5GR27637XNRJ5BVJRMQ7ZE7JGINDMN2FWRNE4PU2WLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427269718 (34h14m25.091218697s) [noise=true , address=145.79.227.18:4006 , piece_id=PWOPK6NWBU65ALSRYYYMSHFLLASY7HWZ7ZU6JWMVXJA2A2ODW3MA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427363119 (34h14m25.0913121s) [noise=true , address=145.79.227.17:4970 , piece_id=PFVWHI4A232TZ6BJ34DWUIU6C3HJ2P4QUO3W6HR6HO226PZDGBFA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.427761264 (34h14m25.091710245s) [noise=true , address=145.79.227.28:4726 , piece_id=6E4SVKWO53YDG3D5BNQRW2LTNVC5MYOFN47EFAIFDDRYZERC5USA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428214622 (34h14m25.092163602s) [noise=true , address=145.79.227.11:4730 , piece_id=L5SJHSIOEWYZAPFMPKVL23JVBQAGMPCIRFXK35H6H7KDRQAR3MVA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428234634 (34h14m25.092183614s) [noise=true , address=145.79.227.32:4836 , piece_id=RSUWJNW3BGE7YW3ME6XPDLHJNYOXRQUJU66RZQNE72W6654JYVKA, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428729183 (34h14m25.092678166s) [noise=true , address=145.79.227.32:4835 , piece_id=IGQPQHZHWZXIWRCSMUXWCBNEPQUXNMTERZNEIGXBE5OQ27757NYQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.428832310 (34h14m25.092781292s) [noise=true , address=145.79.227.15:4874 , piece_id=ZKSWDWF5DATEA2HOGKHEJX6YAHHFN746Y6WBX77L5KQFU3ZHWMUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.429592128 (34h14m25.093541127s) [noise=true , address=145.79.227.27:4691 , piece_id=HPGGSVTUZR2KJTZN56YCYBY7WPRGAEDZUBHVISGMP3CCYIA7QK3A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:53:54.447797690 (34h14m25.111746672s) [noise=true , address=145.79.227.10:4401 , piece_id=6ZCAWDZMPKZ6AJJUZPO6YDGYMQPPQYX5QDPDLLTYNHE27T7EUJVQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: <nil> Oct 31 21:54:24.282494056 (34h14m54.946443036s) [noise=true , address=45.91.116.4:4940 , piece_id=T4IRJYUXZE4AM6WQCQJWV7OWR7OIPCG6CABDT6AD6BW6ZRBSYV7A, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: read tcp 159.223.143.39:35840->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282509279 (34h14m54.946458269s) [noise=true , address=77.90.31.12:4590 , piece_id=XYIQ4ZXHC4F5LKF24D7BFKPRDAB4QBD4ZOCWJIZ4LND7QCEY3HLQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: read tcp 159.223.143.39:36998->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282717367 (34h14m54.946666355s) [noise=true , address=77.90.31.19:4842 , piece_id=SAV3426VO6OMOTNVBZ6C5776YM3ZJA3T6DRFCAK434IUCQA6TJRQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 159.223.143.39:55516->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282833181 (34h14m54.946782153s) [noise=true , address=77.90.31.16:4734 , piece_id=ZF2F7VF5WZFWC46B75L3I2CLLIIPE5QDSJ4ZEIWUX6SPLODUOOUQ, seg_pos={0 0}, upload=35593]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 159.223.143.39:35952->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:54:24.282879533 (34h14m54.946828499s) [seg_pos={0 0}, upload=35593]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:54:24.306226453 (34h14m54.970175433s) [upload=35593]: Done uploading segments... Oct 31 21:54:24.306271889 (34h14m54.970220869s) [upload=35593]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.026353718s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............X.......X.....................................X] duration: 5.001466609s trace: https://jaeger.dev.storj.io/trace/49e5f4c0e8587b3d FAIL (2025-10-31 21:54:17 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001481632s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....X..........X............................................] duration: 1.108673502s trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:53:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.108696525s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............X.......X.....................................] duration: 567.638568ms trace: https://jaeger.dev.storj.io/trace/689f32b5a76d000d FAIL (2025-10-31 21:53:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 567.661616ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.....X................................] duration: 417.949812ms trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 21:53:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 417.974896ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.................X......................................] duration: 682.588669ms trace: https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55 FAIL (2025-10-31 21:53:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded ======LOGS====== 825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments... Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments... Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil> Duration: 682.617996ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.....................X.................................] duration: 411.808893ms trace: https://jaeger.dev.storj.io/trace/2a65f5f21f735664 FAIL (2025-10-31 21:53:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments... Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment... Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment. Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments... Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments... Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 411.838667ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............X.......X....................................] duration: 506.151327ms trace: https://jaeger.dev.storj.io/trace/689f32b5a76d000d FAIL (2025-10-31 21:53:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 506.19415ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X............X.......X......................................] duration: 445.279702ms trace: https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91 FAIL (2025-10-31 21:53:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 445.371372ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.................X.....................................] duration: 430.747666ms trace: https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55 FAIL (2025-10-31 21:52:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded ======LOGS====== 825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments... Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments... Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil> Duration: 430.776245ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X............X.......X.....................................] duration: 484.202545ms trace: https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91 FAIL (2025-10-31 21:52:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 484.225365ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.....................X................................] duration: 477.684148ms trace: https://jaeger.dev.storj.io/trace/2a65f5f21f735664 FAIL (2025-10-31 21:52:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments... Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment... Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment. Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments... Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments... Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 477.72403ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....X..........X...........................................] duration: 1.494051919s trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:52:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.494077244s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................X.......X...................................] duration: 603.89463ms trace: https://jaeger.dev.storj.io/trace/689f32b5a76d000d FAIL (2025-10-31 21:52:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 603.920121ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.....X...............................] duration: 399.953095ms trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 21:52:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 399.982952ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X............X.......X....................................] duration: 461.923756ms trace: https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91 FAIL (2025-10-31 21:52:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 461.949877ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.................X....................................] duration: 515.963957ms trace: https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55 FAIL (2025-10-31 21:52:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded ======LOGS====== 825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments... Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments... Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil> Duration: 515.992869ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......X..........X..........................................] duration: 624.322988ms trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:52:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 624.354339ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.....X..............................] duration: 394.357144ms trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 21:51:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 394.383097ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.....................X...............................] duration: 604.607848ms trace: https://jaeger.dev.storj.io/trace/2a65f5f21f735664 FAIL (2025-10-31 21:51:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments... Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment... Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment. Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments... Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments... Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 604.630721ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................X.......X..................................] duration: 503.806702ms trace: https://jaeger.dev.storj.io/trace/689f32b5a76d000d FAIL (2025-10-31 21:51:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 503.833681ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X............X.......X...................................] duration: 465.795847ms trace: https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91 FAIL (2025-10-31 21:51:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 465.823361ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......X..........X.........................................] duration: 550.076081ms trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:50:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 550.104804ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.................X...................................] duration: 438.991979ms trace: https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55 FAIL (2025-10-31 21:50:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded ======LOGS====== 825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments... Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments... Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil> Duration: 439.01198ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........X..........X........................................] duration: 586.087865ms trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:50:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 586.119426ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.....................X..............................] duration: 865.647451ms trace: https://jaeger.dev.storj.io/trace/2a65f5f21f735664 FAIL (2025-10-31 21:50:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments... Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment... Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment. Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments... Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments... Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 865.761022ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.................X..................................] duration: 460.511961ms trace: https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55 FAIL (2025-10-31 21:50:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded ======LOGS====== 825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments... Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments... Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil> Duration: 460.727867ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................X.......X.................................] duration: 453.758635ms trace: https://jaeger.dev.storj.io/trace/689f32b5a76d000d FAIL (2025-10-31 21:50:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 453.863541ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X.....X.............................] duration: 385.300353ms trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 21:50:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 385.42781ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X............X.......X..................................] duration: 462.115571ms trace: https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91 FAIL (2025-10-31 21:50:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 462.140681ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................X.......X................................] duration: 503.191983ms trace: https://jaeger.dev.storj.io/trace/689f32b5a76d000d FAIL (2025-10-31 21:49:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 503.213974ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.....................X.............................] duration: 550.788313ms trace: https://jaeger.dev.storj.io/trace/2a65f5f21f735664 FAIL (2025-10-31 21:49:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments... Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment... Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment. Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments... Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments... Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 550.811238ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.................X.................................] duration: 456.822528ms trace: https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55 FAIL (2025-10-31 21:49:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded ======LOGS====== 825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments... Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments... Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil> Duration: 456.850738ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................X.......X...............................] duration: 451.258032ms trace: https://jaeger.dev.storj.io/trace/689f32b5a76d000d FAIL (2025-10-31 21:49:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 451.275438ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.....................X............................] duration: 532.962956ms trace: https://jaeger.dev.storj.io/trace/2a65f5f21f735664 FAIL (2025-10-31 21:49:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments... Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment... Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment. Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments... Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments... Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 532.988229ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........X..........X.......................................] duration: 581.975124ms trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:49:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 582.000379ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.................X................................] duration: 457.308939ms trace: https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55 FAIL (2025-10-31 21:49:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded ======LOGS====== 825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments... Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments... Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil> Duration: 457.333744ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X............X.......X.................................] duration: 507.181385ms trace: https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91 FAIL (2025-10-31 21:49:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 507.221475ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................X.......X..............................] duration: 500.531288ms trace: https://jaeger.dev.storj.io/trace/689f32b5a76d000d FAIL (2025-10-31 21:49:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 500.551604ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.....X............................] duration: 363.073314ms trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 21:49:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 363.10031ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.....................X...........................] duration: 452.403981ms trace: https://jaeger.dev.storj.io/trace/2a65f5f21f735664 FAIL (2025-10-31 21:49:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments... Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment... Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment. Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments... Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments... Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 452.431977ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........X..........X......................................] duration: 570.381845ms trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:49:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 570.418555ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................X.......X.............................] duration: 464.518421ms trace: https://jaeger.dev.storj.io/trace/689f32b5a76d000d FAIL (2025-10-31 21:48:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 464.541733ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X............X.......X................................] duration: 461.055453ms trace: https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91 FAIL (2025-10-31 21:48:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 461.086311ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.................X...............................] duration: 696.036973ms trace: https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55 FAIL (2025-10-31 21:48:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded ======LOGS====== 825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments... Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments... Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil> Duration: 696.065322ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.....................X..........................] duration: 436.947972ms trace: https://jaeger.dev.storj.io/trace/2a65f5f21f735664 FAIL (2025-10-31 21:48:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments... Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment... Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment. Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments... Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments... Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 436.962909ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................X.......X............................] duration: 497.553515ms trace: https://jaeger.dev.storj.io/trace/689f32b5a76d000d FAIL (2025-10-31 21:48:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 497.618937ms core uplink us-select-1 us1-placement-20

1
...
70
71
72
73
74
...
2975