Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [XX.....X.XXX...............X..X......X.XX....X...X..........] duration: 1.937379626s FAIL (2025-07-27 00:15:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.937431686s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 48 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X] duration: 30.026313839s FAIL (2025-07-27 00:15:40 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== HvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: context canceled; context canceled Jul 27 00:14:38.138327738 (2h22m53.919261101s) [noise=true , address=77.90.31.13:4628 , piece_id=ESSVQ5ZQNAGL3BQ63MIXOOFXHNXUTBSDAWJLHSVAYJ7N42AJWAMA, seg_pos={0 0}, upload=2192]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Jul 27 00:14:38.138401784 (2h22m53.919335207s) [seg_pos={0 0}, upload=2192]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:14:38.138543579 (2h22m53.919476956s) [upload=2192]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:14:38.138552384 (2h22m53.919485774s) [upload=2192]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:14:38.183797778 (2h22m53.964731141s) [upload=2192]: Done uploading segments... Jul 27 00:14:38.183821771 (2h22m53.96475515s) [upload=2192]: Upload finished. err: <nil> Jul 27 00:15:11.069923050 (2h23m26.850856419s) [upload=2200]: Starting upload Jul 27 00:15:11.070026510 (2h23m26.850959888s) [upload=2200]: Done starting upload Jul 27 00:15:11.070492456 (2h23m26.851425826s) [upload=2200]: Uploading segments... Jul 27 00:15:11.070640477 (2h23m26.851573846s) [upload=2200]: Got next segment. Inline: false Jul 27 00:15:11.070713871 (2h23m26.851647238s) [upload=2200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:15:11.070720299 (2h23m26.851653666s) [upload=2200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:15:11.100379593 (2h23m26.881312981s) [seg_pos={0 0}, upload=2200]: Begin upload segment... Jul 27 00:15:11.100468459 (2h23m26.881401832s) [seg_pos={0 0}, upload=2200]: Done begin upload segment. Jul 27 00:15:11.100511837 (2h23m26.881445223s) [noise=true , address=77.90.31.15:4698 , piece_id=RRZ2NEHFFADRGEYAZJ24HJWC4ZTYT7WHQRHZBXFOLKIOWB47XSFA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.101213380 (2h23m26.882146772s) [noise=true , address=45.91.116.10:4083 , piece_id=5XQGH6P34NMU6RYJG34BFTNZQ3HDBLBCJ56JDMFRFYAS2CEBC7JQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.101756114 (2h23m26.882689501s) [noise=true , address=77.90.31.17:4772 , piece_id=I3VN5VDHW7ITCNMSD2YJIMKW45IIJBPD6N3IXKOWQAXLXVZCBEEA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.102310793 (2h23m26.883244182s) [noise=true , address=45.91.116.8:4048 , piece_id=RIEC7JHDVU6F43LLB4VCLBBJYCISILGLRQKYGDZLWAUESYYWLXKQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.102869541 (2h23m26.883802929s) [noise=true , address=45.91.116.5:4299 , piece_id=MNF6LU3VUVUYOTOHINLS7G4QEWBPKTVECC54B5XPY5LVEKURB6PQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.103487970 (2h23m26.884421356s) [noise=true , address=45.91.116.12:4227 , piece_id=QSNHRVZ2ZDJQHHAGV6WZ5NNLCOYTRQNOAGWPUELXPD6UZV74T3PQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.104339573 (2h23m26.885272958s) [noise=true , address=45.91.116.11:4117 , piece_id=6M5WDB42MJEE2LDFC4BMUJPMTRY6L4FRARUZIB5FZS3ZYOBODAHA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.104784582 (2h23m26.88571797s) [noise=true , address=104.36.82.16:4295 , piece_id=RXYFDGFYR2OOBJLFESFJPGSXVHTK27WPTUOBOAVNV2XNQQP55GYQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.105280368 (2h23m26.886213753s) [noise=true , address=104.36.82.9:4006 , piece_id=4P7AC5VWY6HSCMHVMSZ2YPJAB5XFM3VMOACPAZK6GBZWTX6IQMPA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.105811202 (2h23m26.88674459s) [noise=true , address=45.91.116.4:4937 , piece_id=GFQDOS5XOQULFLJL37XZ63ZQCDS25ZIPOUIGNNYWLWOD2SHKG3PA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.106292266 (2h23m26.887225653s) [noise=true , address=77.90.31.21:4879 , piece_id=EF3RJGC7D2IY6MVPHZYULXF7KFUW32A62UZSUY7X3FKEYMMGP6DA, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.106744527 (2h23m26.887677913s) [noise=true , address=45.91.116.14:4191 , piece_id=OPPKWLMJFONYSULW3KVT5JSV6BE56AHEIFT5KXXNBJDEA5K4HVCQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.107314415 (2h23m26.888247839s) [noise=true , address=45.91.116.7:4009 , piece_id=F7XPPDTWUZFDVUGRXNXNLUTVI3YBLOTGGDBNBVOR6ZZ3KDONGBUQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.107831701 (2h23m26.888765088s) [noise=true , address=45.91.116.6:4975 , piece_id=SM4GHTOZ4C37UWJAPKZNIXRD72MIQ7WTJLOZB4XVBOA75OCHDWTQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.108403892 (2h23m26.889337277s) [noise=true , address=45.91.116.11:4119 , piece_id=S6WQPHHPHWKIYW4IXRS5NBEKRX5QEDWSFJU7F6HBUQHCRGWXUWFQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.108962091 (2h23m26.889895477s) [noise=true , address=104.36.82.6:4873 , piece_id=5NMV4KSV6MKOZVCHCNC4UTJNL5DDSWI4NAM2GKSYEAKWKW3LARBQ, seg_pos={0 0}, upload=2200]: Uploading piece... Jul 27 00:15:11.247197790 (2h23m27.028131176s) [upload=2200]: Next returned nil segment Jul 27 00:15:11.247222340 (2h23m27.028155726s) [upload=2200]: Waiting for error group managing segments... Jul 27 00:15:11.254374474 (2h23m27.035307841s) [noise=true , address=45.91.116.5:4299 , piece_id=MNF6LU3VUVUYOTOHINLS7G4QEWBPKTVECC54B5XPY5LVEKURB6PQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.255204407 (2h23m27.036137787s) [noise=true , address=45.91.116.11:4117 , piece_id=6M5WDB42MJEE2LDFC4BMUJPMTRY6L4FRARUZIB5FZS3ZYOBODAHA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.255915644 (2h23m27.03684901s) [noise=true , address=45.91.116.10:4083 , piece_id=5XQGH6P34NMU6RYJG34BFTNZQ3HDBLBCJ56JDMFRFYAS2CEBC7JQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.261255849 (2h23m27.042189231s) [noise=true , address=45.91.116.7:4009 , piece_id=F7XPPDTWUZFDVUGRXNXNLUTVI3YBLOTGGDBNBVOR6ZZ3KDONGBUQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.262320390 (2h23m27.043253772s) [noise=true , address=45.91.116.6:4975 , piece_id=SM4GHTOZ4C37UWJAPKZNIXRD72MIQ7WTJLOZB4XVBOA75OCHDWTQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.263320002 (2h23m27.044253386s) [noise=true , address=45.91.116.4:4937 , piece_id=GFQDOS5XOQULFLJL37XZ63ZQCDS25ZIPOUIGNNYWLWOD2SHKG3PA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.264411755 (2h23m27.045345122s) [noise=true , address=45.91.116.8:4048 , piece_id=RIEC7JHDVU6F43LLB4VCLBBJYCISILGLRQKYGDZLWAUESYYWLXKQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.266824126 (2h23m27.047757494s) [noise=true , address=77.90.31.15:4698 , piece_id=RRZ2NEHFFADRGEYAZJ24HJWC4ZTYT7WHQRHZBXFOLKIOWB47XSFA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.273455851 (2h23m27.054389235s) [noise=true , address=45.91.116.12:4227 , piece_id=QSNHRVZ2ZDJQHHAGV6WZ5NNLCOYTRQNOAGWPUELXPD6UZV74T3PQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.274219743 (2h23m27.055153106s) [noise=true , address=45.91.116.11:4119 , piece_id=S6WQPHHPHWKIYW4IXRS5NBEKRX5QEDWSFJU7F6HBUQHCRGWXUWFQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:11.326352475 (2h23m27.107285859s) [noise=true , address=45.91.116.14:4191 , piece_id=OPPKWLMJFONYSULW3KVT5JSV6BE56AHEIFT5KXXNBJDEA5K4HVCQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:12.569914903 (2h23m28.350848282s) [noise=true , address=77.90.31.17:4772 , piece_id=I3VN5VDHW7ITCNMSD2YJIMKW45IIJBPD6N3IXKOWQAXLXVZCBEEA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: <nil> Jul 27 00:15:40.941049891 (2h23m56.721983267s) [noise=true , address=77.90.31.21:4879 , piece_id=EF3RJGC7D2IY6MVPHZYULXF7KFUW32A62UZSUY7X3FKEYMMGP6DA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941145501 (2h23m56.722078876s) [noise=true , address=104.36.82.6:4873 , piece_id=5NMV4KSV6MKOZVCHCNC4UTJNL5DDSWI4NAM2GKSYEAKWKW3LARBQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941327898 (2h23m56.722261273s) [noise=true , address=104.36.82.9:4006 , piece_id=4P7AC5VWY6HSCMHVMSZ2YPJAB5XFM3VMOACPAZK6GBZWTX6IQMPA, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941498465 (2h23m56.722431841s) [noise=true , address=104.36.82.16:4295 , piece_id=RXYFDGFYR2OOBJLFESFJPGSXVHTK27WPTUOBOAVNV2XNQQP55GYQ, seg_pos={0 0}, upload=2200]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:50230->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:40.941560349 (2h23m56.722493713s) [seg_pos={0 0}, upload=2200]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:15:40.965136458 (2h23m56.746069842s) [upload=2200]: Done uploading segments... Jul 27 00:15:40.965163512 (2h23m56.746096888s) [upload=2200]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.026336629s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 44 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...] duration: 1.837549174s FAIL (2025-07-27 00:15:26 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 1.837601677s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.027212135s FAIL (2025-07-27 00:15:25 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 25.114292822 (2h22m40.895226199s) [upload=2189]: Uploading segments... Jul 27 00:14:25.114499427 (2h22m40.895432812s) [upload=2189]: Got next segment. Inline: false Jul 27 00:14:25.114563626 (2h22m40.895496989s) [upload=2189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:14:25.114569609 (2h22m40.895502972s) [upload=2189]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:14:25.150755361 (2h22m40.931688751s) [seg_pos={0 0}, upload=2189]: Begin upload segment... Jul 27 00:14:25.150820002 (2h22m40.931753367s) [seg_pos={0 0}, upload=2189]: Done begin upload segment. Jul 27 00:14:25.150844952 (2h22m40.93177833s) [noise=true , address=77.90.31.15:4714 , piece_id=GER5UP2KHUDC6OTJVQF7F2MAO6R4L3QRM5B7YEFEJKLRLBGXFMJA, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.151095640 (2h22m40.932029008s) [noise=true , address=45.91.116.9:4277 , piece_id=RYTK6HJWKERZ6LCEL2ML4KOEFKYGKMVPUJMJ2U5H52CEHEFBPKYQ, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.151491664 (2h22m40.932425033s) [noise=true , address=104.36.82.47:4520 , piece_id=XSPVKBJQP3F7YOD5PLKD7TT26ITLFP4XGAAUYI7M2JLYSTL7BIQA, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.155895216 (2h22m40.936828584s) [noise=true , address=77.90.31.16:4767 , piece_id=E2GI52NZ4LYOD4OPLGIRXROP6OHSPZSF2NAL4LU4FO664S6GPH6Q, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.156369303 (2h22m40.937302671s) [noise=true , address=104.36.82.52:4085 , piece_id=XB2XNBIDJYWEN3L5SOYDCNCTNOSAF6QBTNHNB3KFOYXIGOVK67CA, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.156693469 (2h22m40.937626838s) [noise=true , address=104.36.82.56:4256 , piece_id=CJS7S3GFCJ7CPKEGFLKYZ3NKFZ5VA5S5RVXEYSAF5GFSZGLED24A, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.157084137 (2h22m40.938017507s) [noise=true , address=79.127.220.101:4048 , piece_id=5UY4WS2KCXEVKJTTF4OPQVOXHCSHWRO4A42NDLSZ24763PAF5H4A, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.157474635 (2h22m40.938408011s) [noise=true , address=79.127.205.245:4251 , piece_id=3GX6BDN2WUR4O6SOS5DWM5SPP2JVW2QCCC3RDWDA3OQEEA3JIEYQ, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.157815005 (2h22m40.938748374s) [noise=true , address=79.127.205.235:4685 , piece_id=YAWDYVM6GCHFREGFF4SQQOEM5PDR3EZ5LYHER6373C452KLMT5QA, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.158269401 (2h22m40.939202765s) [noise=true , address=156.146.43.225:4448 , piece_id=YRKDOTWU4PPVCVF7G4N2O66CNKGCQZ3XTVXIHCONDWIDT2S3EAMQ, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.159559710 (2h22m40.940493078s) [noise=true , address=79.127.219.38:4010 , piece_id=R65EBXZP6IKVZPGMERDQWQSW2SAOMTCTP5JER55QAVGAFZUXIT6A, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.160620612 (2h22m40.941553973s) [noise=true , address=77.90.31.21:4906 , piece_id=AM7UFHAGRMVTG4PS63S4TSLHD5BZO6JGQOFO653HBDVOOHIKIABQ, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.163008733 (2h22m40.9439421s) [noise=true , address=156.146.43.225:4449 , piece_id=B2IFELEFV5465S2OHRIIUZ54BINQYYBB7LW365DYTKZEMAKEXWZQ, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.165127857 (2h22m40.946061225s) [noise=true , address=64.49.13.138:4482 , piece_id=SBUMXCEGX4HB62UQ64NTMUES4D2CEC5KD67WZET2PHN456FDPQMA, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.167229434 (2h22m40.948162803s) [noise=true , address=79.127.205.246:4282 , piece_id=23LMGBHFCXME4YKM3BTDH5OQQMLAUZOKDUWXHMRSBYOV2AIHOVFQ, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.169740413 (2h22m40.950673784s) [noise=true , address=79.127.220.97:4361 , piece_id=42ZIAHK6OMVYS5JDHQRV5VRPUG2KGW6PJ4PZBUW5WHPR36HVCWLA, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.171926988 (2h22m40.952860357s) [noise=true , address=104.36.82.54:4178 , piece_id=EIF5VSO7EIRHF57BJ3EO2ALCNK5FWGHC56MQNJ3MHTXQ6LTOTRWQ, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.174056138 (2h22m40.954989507s) [noise=true , address=79.127.201.214:4289 , piece_id=UFPTMNI7QEZ3KVR2IWNAS4GYW7MVK72QYBUANOEH5OS7U5ENIKZA, seg_pos={0 0}, upload=2189]: Uploading piece... Jul 27 00:14:25.233441813 (2h22m41.014375175s) [upload=2189]: Next returned nil segment Jul 27 00:14:25.233464793 (2h22m41.014398156s) [upload=2189]: Waiting for error group managing segments... Jul 27 00:14:25.253316208 (2h22m41.034249571s) [noise=true , address=79.127.219.38:4010 , piece_id=R65EBXZP6IKVZPGMERDQWQSW2SAOMTCTP5JER55QAVGAFZUXIT6A, seg_pos={0 0}, upload=2189]: Done uploading piece. err: <nil> Jul 27 00:14:25.283253681 (2h22m41.064187054s) [noise=true , address=77.90.31.15:4714 , piece_id=GER5UP2KHUDC6OTJVQF7F2MAO6R4L3QRM5B7YEFEJKLRLBGXFMJA, seg_pos={0 0}, upload=2189]: Done uploading piece. err: <nil> Jul 27 00:14:25.301891412 (2h22m41.08282479s) [noise=true , address=77.90.31.16:4767 , piece_id=E2GI52NZ4LYOD4OPLGIRXROP6OHSPZSF2NAL4LU4FO664S6GPH6Q, seg_pos={0 0}, upload=2189]: Done uploading piece. err: <nil> Jul 27 00:14:25.662617715 (2h22m41.443551081s) [noise=true , address=79.127.220.101:4048 , piece_id=5UY4WS2KCXEVKJTTF4OPQVOXHCSHWRO4A42NDLSZ24763PAF5H4A, seg_pos={0 0}, upload=2189]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 27 00:14:25.766214396 (2h22m41.547147765s) [noise=true , address=79.127.205.246:4282 , piece_id=23LMGBHFCXME4YKM3BTDH5OQQMLAUZOKDUWXHMRSBYOV2AIHOVFQ, seg_pos={0 0}, upload=2189]: Done uploading piece. err: <nil> Jul 27 00:14:25.831904084 (2h22m41.612837445s) [noise=true , address=79.127.205.245:4251 , piece_id=3GX6BDN2WUR4O6SOS5DWM5SPP2JVW2QCCC3RDWDA3OQEEA3JIEYQ, seg_pos={0 0}, upload=2189]: Done uploading piece. err: <nil> Jul 27 00:14:26.008447440 (2h22m41.789380824s) [noise=true , address=79.127.205.235:4685 , piece_id=YAWDYVM6GCHFREGFF4SQQOEM5PDR3EZ5LYHER6373C452KLMT5QA, seg_pos={0 0}, upload=2189]: Done uploading piece. err: <nil> Jul 27 00:14:28.219396960 (2h22m44.000330336s) [noise=true , address=79.127.220.97:4361 , piece_id=42ZIAHK6OMVYS5JDHQRV5VRPUG2KGW6PJ4PZBUW5WHPR36HVCWLA, seg_pos={0 0}, upload=2189]: Done uploading piece. err: <nil> Jul 27 00:14:29.654928227 (2h22m45.435861596s) [noise=true , address=156.146.43.225:4449 , piece_id=B2IFELEFV5465S2OHRIIUZ54BINQYYBB7LW365DYTKZEMAKEXWZQ, seg_pos={0 0}, upload=2189]: Done uploading piece. err: <nil> Jul 27 00:14:30.560049952 (2h22m46.340983321s) [noise=true , address=156.146.43.225:4448 , piece_id=YRKDOTWU4PPVCVF7G4N2O66CNKGCQZ3XTVXIHCONDWIDT2S3EAMQ, seg_pos={0 0}, upload=2189]: Done uploading piece. err: <nil> Jul 27 00:14:31.339084597 (2h22m47.120017973s) [noise=true , address=45.91.116.9:4277 , piece_id=RYTK6HJWKERZ6LCEL2ML4KOEFKYGKMVPUJMJ2U5H52CEHEFBPKYQ, seg_pos={0 0}, upload=2189]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 27 00:14:33.405127699 (2h22m49.186061067s) [noise=true , address=79.127.201.214:4289 , piece_id=UFPTMNI7QEZ3KVR2IWNAS4GYW7MVK72QYBUANOEH5OS7U5ENIKZA, seg_pos={0 0}, upload=2189]: Done uploading piece. err: <nil> Jul 27 00:14:35.816824455 (2h22m51.597757822s) [noise=true , address=64.49.13.138:4482 , piece_id=SBUMXCEGX4HB62UQ64NTMUES4D2CEC5KD67WZET2PHN456FDPQMA, seg_pos={0 0}, upload=2189]: Done uploading piece. err: <nil> Jul 27 00:15:25.077942899 (2h23m40.858876279s) [noise=true , address=77.90.31.21:4906 , piece_id=AM7UFHAGRMVTG4PS63S4TSLHD5BZO6JGQOFO653HBDVOOHIKIABQ, seg_pos={0 0}, upload=2189]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:25.077991961 (2h23m40.858925337s) [noise=true , address=104.36.82.56:4256 , piece_id=CJS7S3GFCJ7CPKEGFLKYZ3NKFZ5VA5S5RVXEYSAF5GFSZGLED24A, seg_pos={0 0}, upload=2189]: Done uploading piece. err: ecclient: upload failed (node:11N1BR6rgiYye7CnuM5xDJogUXoUp4FgXrXM41GdhfrRCbbgT8, address:104.36.82.56:4256): protocol: rpc: dial tcp 104.36.82.56:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:25.078128958 (2h23m40.85906234s) [noise=true , address=104.36.82.54:4178 , piece_id=EIF5VSO7EIRHF57BJ3EO2ALCNK5FWGHC56MQNJ3MHTXQ6LTOTRWQ, seg_pos={0 0}, upload=2189]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:104.36.82.54:4178): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:15:25.078259537 (2h23m40.859192912s) [noise=true , address=104.36.82.47:4520 , piece_id=XSPVKBJQP3F7YOD5PLKD7TT26ITLFP4XGAAUYI7M2JLYSTL7BIQA, seg_pos={0 0}, upload=2189]: Done uploading piece. err: ecclient: upload failed (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy, address:104.36.82.47:4520): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:15:25.078446789 (2h23m40.859380154s) [noise=true , address=104.36.82.52:4085 , piece_id=XB2XNBIDJYWEN3L5SOYDCNCTNOSAF6QBTNHNB3KFOYXIGOVK67CA, seg_pos={0 0}, upload=2189]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:104.36.82.52:4085): protocol: rpc: dial tcp 104.36.82.52:4085: i/o timeout; conn closed: op Jul 27 00:15:25.078478910 (2h23m40.859412285s) [seg_pos={0 0}, upload=2189]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 00:15:25.101754143 (2h23m40.882687512s) [upload=2189]: Done uploading segments... Jul 27 00:15:25.101797259 (2h23m40.882730635s) [upload=2189]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.02724214s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X..] duration: 775.871515ms FAIL (2025-07-27 00:15:04 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 775.905699ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.XX.....X.XXX...............X..X......X.XX....X...X.........] duration: 1.982151706s FAIL (2025-07-27 00:14:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.982179014s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 48 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...] duration: 1.783408113s FAIL (2025-07-27 00:14:38 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ddress=104.36.82.6:4875 , piece_id=GHIW37HRIVD6GT3SFE5NG37KJTJPMTXIPZ3DFBD2FZIPDZTWMHBA, seg_pos={0 0}, upload=2158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 159.223.143.39:51714->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:40.438474259 (2h20m56.219407638s) [seg_pos={0 0}, upload=2158]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:12:40.438560347 (2h20m56.21949373s) [upload=2158]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:12:40.438566838 (2h20m56.219500227s) [upload=2158]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:12:40.509422921 (2h20m56.290356315s) [upload=2158]: Done uploading segments... Jul 27 00:12:40.509464315 (2h20m56.290397697s) [upload=2158]: Upload finished. err: <nil> Jul 27 00:13:11.277010948 (2h21m27.057944326s) [upload=2169]: Starting upload Jul 27 00:13:11.277116731 (2h21m27.058050112s) [upload=2169]: Done starting upload Jul 27 00:13:11.277197399 (2h21m27.058130775s) [upload=2169]: Uploading segments... Jul 27 00:13:11.277594524 (2h21m27.058527912s) [upload=2169]: Got next segment. Inline: false Jul 27 00:13:11.277672198 (2h21m27.058605573s) [upload=2169]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:13:11.277679743 (2h21m27.05861311s) [upload=2169]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:13:11.313276916 (2h21m27.094210302s) [seg_pos={0 0}, upload=2169]: Begin upload segment... Jul 27 00:13:11.313357802 (2h21m27.09429119s) [seg_pos={0 0}, upload=2169]: Done begin upload segment. Jul 27 00:13:11.313377828 (2h21m27.094311218s) [noise=true , address=45.91.116.5:4299 , piece_id=SVY5VQZOXDDBRYBR6F3GZVTN2U2P3V5XEVIKGMTPFSKHHTTGHCBA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313399798 (2h21m27.094333185s) [noise=true , address=45.91.116.14:4191 , piece_id=DKXQ24HLMYCNABL2WZXL5UTWA5KEOS6YTOJBOQOS5OXIHXYNZBWQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313517824 (2h21m27.09445121s) [noise=true , address=104.36.82.7:4909 , piece_id=GBAB3QNPQNYRZJZC4NCVYWAIITLQRLYIVO6IW5QJ2LA4TPYNVSVA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313934720 (2h21m27.094868105s) [noise=true , address=77.90.31.19:4842 , piece_id=WQ2GNUFNPH7G4UQECIOPQQ4K5HOUWSZKS35SE4PTK7Y26AOPA2EA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313964599 (2h21m27.094897984s) [noise=true , address=77.90.31.16:4733 , piece_id=I6UAOCUNR5KK35UKYYWY47CIZD7GZMRHAYS5QLFUZWHSED4NXILQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.314326344 (2h21m27.095259726s) [noise=true , address=104.36.82.15:4257 , piece_id=4EEEMA3HQLS6WI25OF7NWWJH5M2TMVEEUFL7TDUZ2KXDQIJKVIMQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.314429730 (2h21m27.095363118s) [noise=true , address=45.91.116.8:4045 , piece_id=V3IOJG5S7SZD725R7S22YOOF7VEKO2LA44EW3WTF7DUZE67OACHQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.314882565 (2h21m27.095815949s) [noise=true , address=77.90.31.15:4699 , piece_id=5JDLEXIEI3GZMRGMMGQ7FHHNFB3ZBJ32SHNZ2VNNFROJNBZYALCQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.315377830 (2h21m27.096311218s) [noise=true , address=45.91.116.6:4974 , piece_id=CRWDOM4F7CPPXU3VD3HATLB2LGO5Q2ABZPGMR5TTKHVPX55VVROQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.315696091 (2h21m27.096629479s) [noise=true , address=45.91.116.12:4226 , piece_id=NBX4ULNVNJ6VZ3X3K5P57QNS52K2QSEA7LOXBM73SCHPEW344QQA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.316218266 (2h21m27.097151651s) [noise=true , address=45.91.116.11:4119 , piece_id=N3VPTFS3Q2MZSKT53MQLP7VR6R37V6TZ5LHDLJPP4ITAKQFD6MTQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.316632666 (2h21m27.097566057s) [noise=true , address=104.36.82.4:4801 , piece_id=IEXEXGC2OSYX6TFQ7ELEYSBAG4T2UPH4RJ2VC5QD47RYQNFORFZA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.316810856 (2h21m27.097744245s) [noise=true , address=45.91.116.4:4939 , piece_id=GX4K5CDRJJRZ6UMBVK2GGYIHUJLDQVYB4D63PCCKPEM42DGKOVTA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.317278579 (2h21m27.09821197s) [noise=true , address=45.91.116.9:4263 , piece_id=3AYY7VIOP44PCPN5IBNRYPL7CGOP6SLBHCZBNIK7HA7DBVQW4R4Q, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.317748179 (2h21m27.098681568s) [noise=true , address=77.90.31.21:4878 , piece_id=NOVSKD6JTPKZV2HNFOMLPST2XBWK4VHANKEDOBEA23HFPHYQJM3Q, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.318152255 (2h21m27.099085649s) [noise=true , address=45.91.116.4:4940 , piece_id=ORVKXYTD3TSCNJ4ZY5KGF3Y6DOCXII2C4M3NRZ3XRNVHZEM4WZWA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.427694395 (2h21m27.208627777s) [upload=2169]: Next returned nil segment Jul 27 00:13:11.427714857 (2h21m27.208648243s) [upload=2169]: Waiting for error group managing segments... Jul 27 00:13:11.446118534 (2h21m27.227051975s) [noise=true , address=45.91.116.11:4119 , piece_id=N3VPTFS3Q2MZSKT53MQLP7VR6R37V6TZ5LHDLJPP4ITAKQFD6MTQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.446797332 (2h21m27.227730717s) [noise=true , address=77.90.31.16:4733 , piece_id=I6UAOCUNR5KK35UKYYWY47CIZD7GZMRHAYS5QLFUZWHSED4NXILQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.447424597 (2h21m27.22835796s) [noise=true , address=45.91.116.6:4974 , piece_id=CRWDOM4F7CPPXU3VD3HATLB2LGO5Q2ABZPGMR5TTKHVPX55VVROQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.447949057 (2h21m27.22888242s) [noise=true , address=77.90.31.15:4699 , piece_id=5JDLEXIEI3GZMRGMMGQ7FHHNFB3ZBJ32SHNZ2VNNFROJNBZYALCQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.448485189 (2h21m27.229418576s) [noise=true , address=45.91.116.4:4939 , piece_id=GX4K5CDRJJRZ6UMBVK2GGYIHUJLDQVYB4D63PCCKPEM42DGKOVTA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.449018483 (2h21m27.229951844s) [noise=true , address=45.91.116.9:4263 , piece_id=3AYY7VIOP44PCPN5IBNRYPL7CGOP6SLBHCZBNIK7HA7DBVQW4R4Q, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.449709382 (2h21m27.23064276s) [noise=true , address=45.91.116.8:4045 , piece_id=V3IOJG5S7SZD725R7S22YOOF7VEKO2LA44EW3WTF7DUZE67OACHQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.449742304 (2h21m27.230675678s) [noise=true , address=45.91.116.5:4299 , piece_id=SVY5VQZOXDDBRYBR6F3GZVTN2U2P3V5XEVIKGMTPFSKHHTTGHCBA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.450341755 (2h21m27.231275132s) [noise=true , address=45.91.116.4:4940 , piece_id=ORVKXYTD3TSCNJ4ZY5KGF3Y6DOCXII2C4M3NRZ3XRNVHZEM4WZWA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.450449996 (2h21m27.231383369s) [noise=true , address=45.91.116.12:4226 , piece_id=NBX4ULNVNJ6VZ3X3K5P57QNS52K2QSEA7LOXBM73SCHPEW344QQA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.692726076 (2h21m27.473659458s) [noise=true , address=45.91.116.14:4191 , piece_id=DKXQ24HLMYCNABL2WZXL5UTWA5KEOS6YTOJBOQOS5OXIHXYNZBWQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:12.789383367 (2h21m28.570316735s) [noise=true , address=77.90.31.19:4842 , piece_id=WQ2GNUFNPH7G4UQECIOPQQ4K5HOUWSZKS35SE4PTK7Y26AOPA2EA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:41.236943285 (2h21m57.017876661s) [noise=true , address=104.36.82.7:4909 , piece_id=GBAB3QNPQNYRZJZC4NCVYWAIITLQRLYIVO6IW5QJ2LA4TPYNVSVA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.236967972 (2h21m57.017901341s) [noise=true , address=104.36.82.15:4257 , piece_id=4EEEMA3HQLS6WI25OF7NWWJH5M2TMVEEUFL7TDUZ2KXDQIJKVIMQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 159.223.143.39:51340->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.236995526 (2h21m57.017928895s) [noise=true , address=104.36.82.4:4801 , piece_id=IEXEXGC2OSYX6TFQ7ELEYSBAG4T2UPH4RJ2VC5QD47RYQNFORFZA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.237035393 (2h21m57.01796877s) [noise=true , address=77.90.31.21:4878 , piece_id=NOVSKD6JTPKZV2HNFOMLPST2XBWK4VHANKEDOBEA23HFPHYQJM3Q, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:49578->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.237062045 (2h21m57.017995421s) [seg_pos={0 0}, upload=2169]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:13:41.260380631 (2h21m57.041313999s) [upload=2169]: Done uploading segments... Jul 27 00:13:41.260406109 (2h21m57.041339474s) [upload=2169]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.783468183s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [....XX..............X.X........X...XX.......................] duration: 14.468555081s FAIL (2025-07-27 00:14:26 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.468596088s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X..] duration: 1.912088458s FAIL (2025-07-27 00:14:14 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ddress=104.36.82.6:4875 , piece_id=GHIW37HRIVD6GT3SFE5NG37KJTJPMTXIPZ3DFBD2FZIPDZTWMHBA, seg_pos={0 0}, upload=2158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 159.223.143.39:51714->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:40.438474259 (2h20m56.219407638s) [seg_pos={0 0}, upload=2158]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:12:40.438560347 (2h20m56.21949373s) [upload=2158]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:12:40.438566838 (2h20m56.219500227s) [upload=2158]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:12:40.509422921 (2h20m56.290356315s) [upload=2158]: Done uploading segments... Jul 27 00:12:40.509464315 (2h20m56.290397697s) [upload=2158]: Upload finished. err: <nil> Jul 27 00:13:11.277010948 (2h21m27.057944326s) [upload=2169]: Starting upload Jul 27 00:13:11.277116731 (2h21m27.058050112s) [upload=2169]: Done starting upload Jul 27 00:13:11.277197399 (2h21m27.058130775s) [upload=2169]: Uploading segments... Jul 27 00:13:11.277594524 (2h21m27.058527912s) [upload=2169]: Got next segment. Inline: false Jul 27 00:13:11.277672198 (2h21m27.058605573s) [upload=2169]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:13:11.277679743 (2h21m27.05861311s) [upload=2169]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:13:11.313276916 (2h21m27.094210302s) [seg_pos={0 0}, upload=2169]: Begin upload segment... Jul 27 00:13:11.313357802 (2h21m27.09429119s) [seg_pos={0 0}, upload=2169]: Done begin upload segment. Jul 27 00:13:11.313377828 (2h21m27.094311218s) [noise=true , address=45.91.116.5:4299 , piece_id=SVY5VQZOXDDBRYBR6F3GZVTN2U2P3V5XEVIKGMTPFSKHHTTGHCBA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313399798 (2h21m27.094333185s) [noise=true , address=45.91.116.14:4191 , piece_id=DKXQ24HLMYCNABL2WZXL5UTWA5KEOS6YTOJBOQOS5OXIHXYNZBWQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313517824 (2h21m27.09445121s) [noise=true , address=104.36.82.7:4909 , piece_id=GBAB3QNPQNYRZJZC4NCVYWAIITLQRLYIVO6IW5QJ2LA4TPYNVSVA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313934720 (2h21m27.094868105s) [noise=true , address=77.90.31.19:4842 , piece_id=WQ2GNUFNPH7G4UQECIOPQQ4K5HOUWSZKS35SE4PTK7Y26AOPA2EA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313964599 (2h21m27.094897984s) [noise=true , address=77.90.31.16:4733 , piece_id=I6UAOCUNR5KK35UKYYWY47CIZD7GZMRHAYS5QLFUZWHSED4NXILQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.314326344 (2h21m27.095259726s) [noise=true , address=104.36.82.15:4257 , piece_id=4EEEMA3HQLS6WI25OF7NWWJH5M2TMVEEUFL7TDUZ2KXDQIJKVIMQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.314429730 (2h21m27.095363118s) [noise=true , address=45.91.116.8:4045 , piece_id=V3IOJG5S7SZD725R7S22YOOF7VEKO2LA44EW3WTF7DUZE67OACHQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.314882565 (2h21m27.095815949s) [noise=true , address=77.90.31.15:4699 , piece_id=5JDLEXIEI3GZMRGMMGQ7FHHNFB3ZBJ32SHNZ2VNNFROJNBZYALCQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.315377830 (2h21m27.096311218s) [noise=true , address=45.91.116.6:4974 , piece_id=CRWDOM4F7CPPXU3VD3HATLB2LGO5Q2ABZPGMR5TTKHVPX55VVROQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.315696091 (2h21m27.096629479s) [noise=true , address=45.91.116.12:4226 , piece_id=NBX4ULNVNJ6VZ3X3K5P57QNS52K2QSEA7LOXBM73SCHPEW344QQA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.316218266 (2h21m27.097151651s) [noise=true , address=45.91.116.11:4119 , piece_id=N3VPTFS3Q2MZSKT53MQLP7VR6R37V6TZ5LHDLJPP4ITAKQFD6MTQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.316632666 (2h21m27.097566057s) [noise=true , address=104.36.82.4:4801 , piece_id=IEXEXGC2OSYX6TFQ7ELEYSBAG4T2UPH4RJ2VC5QD47RYQNFORFZA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.316810856 (2h21m27.097744245s) [noise=true , address=45.91.116.4:4939 , piece_id=GX4K5CDRJJRZ6UMBVK2GGYIHUJLDQVYB4D63PCCKPEM42DGKOVTA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.317278579 (2h21m27.09821197s) [noise=true , address=45.91.116.9:4263 , piece_id=3AYY7VIOP44PCPN5IBNRYPL7CGOP6SLBHCZBNIK7HA7DBVQW4R4Q, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.317748179 (2h21m27.098681568s) [noise=true , address=77.90.31.21:4878 , piece_id=NOVSKD6JTPKZV2HNFOMLPST2XBWK4VHANKEDOBEA23HFPHYQJM3Q, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.318152255 (2h21m27.099085649s) [noise=true , address=45.91.116.4:4940 , piece_id=ORVKXYTD3TSCNJ4ZY5KGF3Y6DOCXII2C4M3NRZ3XRNVHZEM4WZWA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.427694395 (2h21m27.208627777s) [upload=2169]: Next returned nil segment Jul 27 00:13:11.427714857 (2h21m27.208648243s) [upload=2169]: Waiting for error group managing segments... Jul 27 00:13:11.446118534 (2h21m27.227051975s) [noise=true , address=45.91.116.11:4119 , piece_id=N3VPTFS3Q2MZSKT53MQLP7VR6R37V6TZ5LHDLJPP4ITAKQFD6MTQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.446797332 (2h21m27.227730717s) [noise=true , address=77.90.31.16:4733 , piece_id=I6UAOCUNR5KK35UKYYWY47CIZD7GZMRHAYS5QLFUZWHSED4NXILQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.447424597 (2h21m27.22835796s) [noise=true , address=45.91.116.6:4974 , piece_id=CRWDOM4F7CPPXU3VD3HATLB2LGO5Q2ABZPGMR5TTKHVPX55VVROQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.447949057 (2h21m27.22888242s) [noise=true , address=77.90.31.15:4699 , piece_id=5JDLEXIEI3GZMRGMMGQ7FHHNFB3ZBJ32SHNZ2VNNFROJNBZYALCQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.448485189 (2h21m27.229418576s) [noise=true , address=45.91.116.4:4939 , piece_id=GX4K5CDRJJRZ6UMBVK2GGYIHUJLDQVYB4D63PCCKPEM42DGKOVTA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.449018483 (2h21m27.229951844s) [noise=true , address=45.91.116.9:4263 , piece_id=3AYY7VIOP44PCPN5IBNRYPL7CGOP6SLBHCZBNIK7HA7DBVQW4R4Q, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.449709382 (2h21m27.23064276s) [noise=true , address=45.91.116.8:4045 , piece_id=V3IOJG5S7SZD725R7S22YOOF7VEKO2LA44EW3WTF7DUZE67OACHQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.449742304 (2h21m27.230675678s) [noise=true , address=45.91.116.5:4299 , piece_id=SVY5VQZOXDDBRYBR6F3GZVTN2U2P3V5XEVIKGMTPFSKHHTTGHCBA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.450341755 (2h21m27.231275132s) [noise=true , address=45.91.116.4:4940 , piece_id=ORVKXYTD3TSCNJ4ZY5KGF3Y6DOCXII2C4M3NRZ3XRNVHZEM4WZWA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.450449996 (2h21m27.231383369s) [noise=true , address=45.91.116.12:4226 , piece_id=NBX4ULNVNJ6VZ3X3K5P57QNS52K2QSEA7LOXBM73SCHPEW344QQA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.692726076 (2h21m27.473659458s) [noise=true , address=45.91.116.14:4191 , piece_id=DKXQ24HLMYCNABL2WZXL5UTWA5KEOS6YTOJBOQOS5OXIHXYNZBWQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:12.789383367 (2h21m28.570316735s) [noise=true , address=77.90.31.19:4842 , piece_id=WQ2GNUFNPH7G4UQECIOPQQ4K5HOUWSZKS35SE4PTK7Y26AOPA2EA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:41.236943285 (2h21m57.017876661s) [noise=true , address=104.36.82.7:4909 , piece_id=GBAB3QNPQNYRZJZC4NCVYWAIITLQRLYIVO6IW5QJ2LA4TPYNVSVA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.236967972 (2h21m57.017901341s) [noise=true , address=104.36.82.15:4257 , piece_id=4EEEMA3HQLS6WI25OF7NWWJH5M2TMVEEUFL7TDUZ2KXDQIJKVIMQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 159.223.143.39:51340->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.236995526 (2h21m57.017928895s) [noise=true , address=104.36.82.4:4801 , piece_id=IEXEXGC2OSYX6TFQ7ELEYSBAG4T2UPH4RJ2VC5QD47RYQNFORFZA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.237035393 (2h21m57.01796877s) [noise=true , address=77.90.31.21:4878 , piece_id=NOVSKD6JTPKZV2HNFOMLPST2XBWK4VHANKEDOBEA23HFPHYQJM3Q, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:49578->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.237062045 (2h21m57.017995421s) [seg_pos={0 0}, upload=2169]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:13:41.260380631 (2h21m57.041313999s) [upload=2169]: Done uploading segments... Jul 27 00:13:41.260406109 (2h21m57.041339474s) [upload=2169]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.912158471s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.024966463s FAIL (2025-07-27 00:14:11 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.611174432 (2h21m27.392107833s) [noise=true , address=104.36.82.55:4216 , piece_id=B2QBMDBSB2CR5XL2B4M7JGG3YFUD2EWOFV5VVKBDNLKW3ON375NA, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.611503399 (2h21m27.392436789s) [noise=true , address=77.90.31.16:4746 , piece_id=SDHOWJHHDCHWXZJV5VEHJCJYU5ETVIEXWAQAEUO4RZRZJPRDNJDQ, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.611675073 (2h21m27.39260846s) [noise=true , address=156.146.43.225:4448 , piece_id=KBIZ5QKQOLU7F3JBDCYWFXP5NEG73XK5L6A5HXMILKXSX42ZNW7Q, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.612101828 (2h21m27.393035217s) [noise=true , address=79.127.205.245:4250 , piece_id=ODIMS6ZUNZMF5V6NK7JBLTRXMUXOL2C2TYV4LVW7OAJBKSTLYORQ, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.612190737 (2h21m27.393124123s) [noise=true , address=104.36.82.52:4104 , piece_id=XUVOXJCDWGRWDCCO2LNGIX24BROQKDOVJJD4RP4PEO26ITMGLEPA, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.612615908 (2h21m27.393549296s) [noise=true , address=79.127.201.215:4357 , piece_id=4GRR477U3XQ65QMONSBESNRPVWSX3RRRLG5JRVU77BN33D4FPZLQ, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.612902304 (2h21m27.393835691s) [noise=true , address=79.127.201.214:4321 , piece_id=VJJQ6PY7PZM76IU4DQ2BU5EEFG4QGLCZRYAWOAAX7XCNKKXTMRVA, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.613018061 (2h21m27.393951447s) [noise=true , address=104.36.82.2:4740 , piece_id=2CK7IGYFNQN4DEGG5YNHZ6B4ECVCYZX44AEV72ZAQCLWK2F2MFOA, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.613452167 (2h21m27.394385556s) [noise=true , address=104.36.82.46:4503 , piece_id=IT7AHX64ZIOOIFFM62EUELXD7VIHIPUQP7E4JP3UVHKGRR7B6FNQ, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.613905389 (2h21m27.394838815s) [noise=true , address=79.127.201.214:4289 , piece_id=JZZY3QTUDRZYOUQPN6UHB7YZEYYUZCXPXJCRQQLWMHKFKAC3V5LQ, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.614324661 (2h21m27.395258051s) [noise=true , address=64.49.13.26:4953 , piece_id=4US56S2F3JD34WCAZA7AABI6JZHX4ZKF2JB3JMXBCD723XBIQHBQ, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.614479301 (2h21m27.395412692s) [noise=true , address=79.127.205.246:4262 , piece_id=JPRU4ACG7VLQBWB2DH5LIZY5S2QAAH4UYNVKAHFPBGYDRG4ETO3A, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.614731327 (2h21m27.395664721s) [noise=true , address=79.127.219.38:4001 , piece_id=ST4QY2BAH32ADFZOXGJBK2AL62HJKZQ7K57CIH6HOU74FSTKW3OQ, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.614977081 (2h21m27.39591047s) [noise=true , address=104.36.82.42:4829 , piece_id=MWASMHZGME3OJDBOFSGPDOPCSRE2R3HXAVI2UUFNYGMQPOJ22D7A, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.615189572 (2h21m27.396122962s) [noise=true , address=77.90.31.18:4821 , piece_id=C33PAKSCFIWYLKC3ZMX7JSKFTEB4TID5IVKDTGHJGHHHCGL2XEYA, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.615599924 (2h21m27.396533312s) [noise=true , address=77.90.31.19:4875 , piece_id=F7DUM74CQ7EJENYKTIJDSPD72KLK7RADE7EMQG5O5VJBGE5HN7BQ, seg_pos={0 0}, upload=2170]: Uploading piece... Jul 27 00:13:11.718126438 (2h21m27.499059829s) [upload=2170]: Next returned nil segment Jul 27 00:13:11.718148911 (2h21m27.499082299s) [upload=2170]: Waiting for error group managing segments... Jul 27 00:13:11.765404359 (2h21m27.546337736s) [noise=true , address=77.90.31.16:4746 , piece_id=SDHOWJHHDCHWXZJV5VEHJCJYU5ETVIEXWAQAEUO4RZRZJPRDNJDQ, seg_pos={0 0}, upload=2170]: Done uploading piece. err: <nil> Jul 27 00:13:11.766674933 (2h21m27.547608309s) [noise=true , address=156.146.43.225:4448 , piece_id=KBIZ5QKQOLU7F3JBDCYWFXP5NEG73XK5L6A5HXMILKXSX42ZNW7Q, seg_pos={0 0}, upload=2170]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 27 00:13:11.773175838 (2h21m27.554109218s) [noise=true , address=79.127.201.214:4321 , piece_id=VJJQ6PY7PZM76IU4DQ2BU5EEFG4QGLCZRYAWOAAX7XCNKKXTMRVA, seg_pos={0 0}, upload=2170]: Done uploading piece. err: ecclient: upload failed (node:127EgxjVRhAGcz1DJjE539rWnmDjsVTRFQc1esEZUyQ1yxGiCD5, address:79.127.201.214:4321): protocol: storage node overloaded, request limit: 2000 Jul 27 00:13:11.803170204 (2h21m27.58410359s) [noise=true , address=79.127.201.214:4289 , piece_id=JZZY3QTUDRZYOUQPN6UHB7YZEYYUZCXPXJCRQQLWMHKFKAC3V5LQ, seg_pos={0 0}, upload=2170]: Done uploading piece. err: ecclient: upload failed (node:1YfeEiw7zKAYgUcMqMLgEiS2t1gMqdwrpuAdTb2dZouuXRBSM4, address:79.127.201.214:4289): protocol: storage node overloaded, request limit: 2000 Jul 27 00:13:11.893244332 (2h21m27.674177702s) [noise=true , address=79.127.219.38:4001 , piece_id=ST4QY2BAH32ADFZOXGJBK2AL62HJKZQ7K57CIH6HOU74FSTKW3OQ, seg_pos={0 0}, upload=2170]: Done uploading piece. err: <nil> Jul 27 00:13:12.197473394 (2h21m27.978406772s) [noise=true , address=79.127.205.245:4250 , piece_id=ODIMS6ZUNZMF5V6NK7JBLTRXMUXOL2C2TYV4LVW7OAJBKSTLYORQ, seg_pos={0 0}, upload=2170]: Done uploading piece. err: <nil> Jul 27 00:13:12.240826317 (2h21m28.0217597s) [noise=true , address=79.127.205.246:4262 , piece_id=JPRU4ACG7VLQBWB2DH5LIZY5S2QAAH4UYNVKAHFPBGYDRG4ETO3A, seg_pos={0 0}, upload=2170]: Done uploading piece. err: <nil> Jul 27 00:13:12.726449912 (2h21m28.507383275s) [noise=true , address=79.127.205.233:4650 , piece_id=MRO5FS7MCY7SVNDT57VAAQ6WEIDUZIDJGUVS2DWV7LCI76OZOC4Q, seg_pos={0 0}, upload=2170]: Done uploading piece. err: <nil> Jul 27 00:13:14.349472359 (2h21m30.130405736s) [noise=true , address=64.49.13.26:4953 , piece_id=4US56S2F3JD34WCAZA7AABI6JZHX4ZKF2JB3JMXBCD723XBIQHBQ, seg_pos={0 0}, upload=2170]: Done uploading piece. err: ecclient: upload failed (node:12JvztpAXVj8B4XY7FgaFcsWtqwSxzpV4sRc926EigSpGqnytXE, address:64.49.13.26:4953): protocol: storage node overloaded, request limit: 2000 Jul 27 00:13:19.893805564 (2h21m35.674738943s) [noise=true , address=79.127.201.215:4357 , piece_id=4GRR477U3XQ65QMONSBESNRPVWSX3RRRLG5JRVU77BN33D4FPZLQ, seg_pos={0 0}, upload=2170]: Done uploading piece. err: <nil> Jul 27 00:13:29.039662290 (2h21m44.820595666s) [noise=true , address=77.90.31.19:4875 , piece_id=F7DUM74CQ7EJENYKTIJDSPD72KLK7RADE7EMQG5O5VJBGE5HN7BQ, seg_pos={0 0}, upload=2170]: Done uploading piece. err: <nil> Jul 27 00:14:11.536688272 (2h22m27.317621668s) [noise=true , address=104.36.82.46:4503 , piece_id=IT7AHX64ZIOOIFFM62EUELXD7VIHIPUQP7E4JP3UVHKGRR7B6FNQ, seg_pos={0 0}, upload=2170]: Done uploading piece. err: ecclient: upload failed (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm, address:104.36.82.46:4503): protocol: rpc: dial tcp 104.36.82.46:4503: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:14:11.536722994 (2h22m27.317656369s) [noise=true , address=104.36.82.2:4740 , piece_id=2CK7IGYFNQN4DEGG5YNHZ6B4ECVCYZX44AEV72ZAQCLWK2F2MFOA, seg_pos={0 0}, upload=2170]: Done uploading piece. err: ecclient: upload failed (node:12wgQDbE17gP8c6kbKnNyywSkY8Hk7mchcMaymLBQzPwRsSB8QE, address:104.36.82.2:4740): protocol: rpc: dial tcp 104.36.82.2:4740: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:14:11.536767955 (2h22m27.31770134s) [noise=true , address=104.36.82.47:4550 , piece_id=QFDX5TWB3USUU3G5LFV3SMQFEXXW4RKH6XUREAOOVZUYU3PPNM6A, seg_pos={0 0}, upload=2170]: Done uploading piece. err: ecclient: upload failed (node:1Fz8JANjsHWJkHj7Y2JAkr7NTKqDtcKjRW4kDaYGLJYwgWnxbS, address:104.36.82.47:4550): protocol: rpc: dial tcp 104.36.82.47:4550: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:14:11.536812915 (2h22m27.317746296s) [noise=true , address=104.36.82.52:4104 , piece_id=XUVOXJCDWGRWDCCO2LNGIX24BROQKDOVJJD4RP4PEO26ITMGLEPA, seg_pos={0 0}, upload=2170]: Done uploading piece. err: ecclient: upload failed (node:12qmPpCmXAvuJZn2KR5Kr4nHYwKEzWvwqZ82nqNka5u1mKaZmXs, address:104.36.82.52:4104): protocol: rpc: dial tcp 104.36.82.52:4104: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:14:11.536835230 (2h22m27.317768611s) [noise=true , address=104.36.82.55:4216 , piece_id=B2QBMDBSB2CR5XL2B4M7JGG3YFUD2EWOFV5VVKBDNLKW3ON375NA, seg_pos={0 0}, upload=2170]: Done uploading piece. err: ecclient: upload failed (node:12vYNo8tVKDfSphJzsryTLhezUDVw3DA1NaK7ZXcXYcEw4pf1Pt, address:104.36.82.55:4216): protocol: rpc: dial tcp 104.36.82.55:4216: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:14:11.536884672 (2h22m27.317818049s) [noise=true , address=104.36.82.42:4829 , piece_id=MWASMHZGME3OJDBOFSGPDOPCSRE2R3HXAVI2UUFNYGMQPOJ22D7A, seg_pos={0 0}, upload=2170]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: rpc: dial tcp 104.36.82.42:4829: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:14:11.536916240 (2h22m27.317849615s) [noise=true , address=77.90.31.18:4821 , piece_id=C33PAKSCFIWYLKC3ZMX7JSKFTEB4TID5IVKDTGHJGHHHCGL2XEYA, seg_pos={0 0}, upload=2170]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: write tcp 159.223.143.39:44632->77.90.31.18:4821: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:14:11.536944871 (2h22m27.317878245s) [seg_pos={0 0}, upload=2170]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 27 00:14:11.559077772 (2h22m27.340011149s) [upload=2170]: Done uploading segments... Jul 27 00:14:11.559109021 (2h22m27.340042399s) [upload=2170]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.024987427s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X.] duration: 592.997284ms FAIL (2025-07-27 00:14:06 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 593.02588ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X] duration: 30.038140627s FAIL (2025-07-27 00:14:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.038166621s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [X....XX..............X.X........X...XX......................] duration: 29.376343844s FAIL (2025-07-27 00:13:56 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.376387289s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..XX.....X.XXX...............X..X......X.XX....X...X........] duration: 887.946534ms FAIL (2025-07-27 00:13:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 888.05571ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 50 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X.] duration: 596.519081ms FAIL (2025-07-27 00:13:43 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ddress=104.36.82.6:4875 , piece_id=GHIW37HRIVD6GT3SFE5NG37KJTJPMTXIPZ3DFBD2FZIPDZTWMHBA, seg_pos={0 0}, upload=2158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 159.223.143.39:51714->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:40.438474259 (2h20m56.219407638s) [seg_pos={0 0}, upload=2158]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:12:40.438560347 (2h20m56.21949373s) [upload=2158]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:12:40.438566838 (2h20m56.219500227s) [upload=2158]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:12:40.509422921 (2h20m56.290356315s) [upload=2158]: Done uploading segments... Jul 27 00:12:40.509464315 (2h20m56.290397697s) [upload=2158]: Upload finished. err: <nil> Jul 27 00:13:11.277010948 (2h21m27.057944326s) [upload=2169]: Starting upload Jul 27 00:13:11.277116731 (2h21m27.058050112s) [upload=2169]: Done starting upload Jul 27 00:13:11.277197399 (2h21m27.058130775s) [upload=2169]: Uploading segments... Jul 27 00:13:11.277594524 (2h21m27.058527912s) [upload=2169]: Got next segment. Inline: false Jul 27 00:13:11.277672198 (2h21m27.058605573s) [upload=2169]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:13:11.277679743 (2h21m27.05861311s) [upload=2169]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:13:11.313276916 (2h21m27.094210302s) [seg_pos={0 0}, upload=2169]: Begin upload segment... Jul 27 00:13:11.313357802 (2h21m27.09429119s) [seg_pos={0 0}, upload=2169]: Done begin upload segment. Jul 27 00:13:11.313377828 (2h21m27.094311218s) [noise=true , address=45.91.116.5:4299 , piece_id=SVY5VQZOXDDBRYBR6F3GZVTN2U2P3V5XEVIKGMTPFSKHHTTGHCBA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313399798 (2h21m27.094333185s) [noise=true , address=45.91.116.14:4191 , piece_id=DKXQ24HLMYCNABL2WZXL5UTWA5KEOS6YTOJBOQOS5OXIHXYNZBWQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313517824 (2h21m27.09445121s) [noise=true , address=104.36.82.7:4909 , piece_id=GBAB3QNPQNYRZJZC4NCVYWAIITLQRLYIVO6IW5QJ2LA4TPYNVSVA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313934720 (2h21m27.094868105s) [noise=true , address=77.90.31.19:4842 , piece_id=WQ2GNUFNPH7G4UQECIOPQQ4K5HOUWSZKS35SE4PTK7Y26AOPA2EA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313964599 (2h21m27.094897984s) [noise=true , address=77.90.31.16:4733 , piece_id=I6UAOCUNR5KK35UKYYWY47CIZD7GZMRHAYS5QLFUZWHSED4NXILQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.314326344 (2h21m27.095259726s) [noise=true , address=104.36.82.15:4257 , piece_id=4EEEMA3HQLS6WI25OF7NWWJH5M2TMVEEUFL7TDUZ2KXDQIJKVIMQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.314429730 (2h21m27.095363118s) [noise=true , address=45.91.116.8:4045 , piece_id=V3IOJG5S7SZD725R7S22YOOF7VEKO2LA44EW3WTF7DUZE67OACHQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.314882565 (2h21m27.095815949s) [noise=true , address=77.90.31.15:4699 , piece_id=5JDLEXIEI3GZMRGMMGQ7FHHNFB3ZBJ32SHNZ2VNNFROJNBZYALCQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.315377830 (2h21m27.096311218s) [noise=true , address=45.91.116.6:4974 , piece_id=CRWDOM4F7CPPXU3VD3HATLB2LGO5Q2ABZPGMR5TTKHVPX55VVROQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.315696091 (2h21m27.096629479s) [noise=true , address=45.91.116.12:4226 , piece_id=NBX4ULNVNJ6VZ3X3K5P57QNS52K2QSEA7LOXBM73SCHPEW344QQA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.316218266 (2h21m27.097151651s) [noise=true , address=45.91.116.11:4119 , piece_id=N3VPTFS3Q2MZSKT53MQLP7VR6R37V6TZ5LHDLJPP4ITAKQFD6MTQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.316632666 (2h21m27.097566057s) [noise=true , address=104.36.82.4:4801 , piece_id=IEXEXGC2OSYX6TFQ7ELEYSBAG4T2UPH4RJ2VC5QD47RYQNFORFZA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.316810856 (2h21m27.097744245s) [noise=true , address=45.91.116.4:4939 , piece_id=GX4K5CDRJJRZ6UMBVK2GGYIHUJLDQVYB4D63PCCKPEM42DGKOVTA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.317278579 (2h21m27.09821197s) [noise=true , address=45.91.116.9:4263 , piece_id=3AYY7VIOP44PCPN5IBNRYPL7CGOP6SLBHCZBNIK7HA7DBVQW4R4Q, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.317748179 (2h21m27.098681568s) [noise=true , address=77.90.31.21:4878 , piece_id=NOVSKD6JTPKZV2HNFOMLPST2XBWK4VHANKEDOBEA23HFPHYQJM3Q, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.318152255 (2h21m27.099085649s) [noise=true , address=45.91.116.4:4940 , piece_id=ORVKXYTD3TSCNJ4ZY5KGF3Y6DOCXII2C4M3NRZ3XRNVHZEM4WZWA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.427694395 (2h21m27.208627777s) [upload=2169]: Next returned nil segment Jul 27 00:13:11.427714857 (2h21m27.208648243s) [upload=2169]: Waiting for error group managing segments... Jul 27 00:13:11.446118534 (2h21m27.227051975s) [noise=true , address=45.91.116.11:4119 , piece_id=N3VPTFS3Q2MZSKT53MQLP7VR6R37V6TZ5LHDLJPP4ITAKQFD6MTQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.446797332 (2h21m27.227730717s) [noise=true , address=77.90.31.16:4733 , piece_id=I6UAOCUNR5KK35UKYYWY47CIZD7GZMRHAYS5QLFUZWHSED4NXILQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.447424597 (2h21m27.22835796s) [noise=true , address=45.91.116.6:4974 , piece_id=CRWDOM4F7CPPXU3VD3HATLB2LGO5Q2ABZPGMR5TTKHVPX55VVROQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.447949057 (2h21m27.22888242s) [noise=true , address=77.90.31.15:4699 , piece_id=5JDLEXIEI3GZMRGMMGQ7FHHNFB3ZBJ32SHNZ2VNNFROJNBZYALCQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.448485189 (2h21m27.229418576s) [noise=true , address=45.91.116.4:4939 , piece_id=GX4K5CDRJJRZ6UMBVK2GGYIHUJLDQVYB4D63PCCKPEM42DGKOVTA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.449018483 (2h21m27.229951844s) [noise=true , address=45.91.116.9:4263 , piece_id=3AYY7VIOP44PCPN5IBNRYPL7CGOP6SLBHCZBNIK7HA7DBVQW4R4Q, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.449709382 (2h21m27.23064276s) [noise=true , address=45.91.116.8:4045 , piece_id=V3IOJG5S7SZD725R7S22YOOF7VEKO2LA44EW3WTF7DUZE67OACHQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.449742304 (2h21m27.230675678s) [noise=true , address=45.91.116.5:4299 , piece_id=SVY5VQZOXDDBRYBR6F3GZVTN2U2P3V5XEVIKGMTPFSKHHTTGHCBA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.450341755 (2h21m27.231275132s) [noise=true , address=45.91.116.4:4940 , piece_id=ORVKXYTD3TSCNJ4ZY5KGF3Y6DOCXII2C4M3NRZ3XRNVHZEM4WZWA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.450449996 (2h21m27.231383369s) [noise=true , address=45.91.116.12:4226 , piece_id=NBX4ULNVNJ6VZ3X3K5P57QNS52K2QSEA7LOXBM73SCHPEW344QQA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.692726076 (2h21m27.473659458s) [noise=true , address=45.91.116.14:4191 , piece_id=DKXQ24HLMYCNABL2WZXL5UTWA5KEOS6YTOJBOQOS5OXIHXYNZBWQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:12.789383367 (2h21m28.570316735s) [noise=true , address=77.90.31.19:4842 , piece_id=WQ2GNUFNPH7G4UQECIOPQQ4K5HOUWSZKS35SE4PTK7Y26AOPA2EA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:41.236943285 (2h21m57.017876661s) [noise=true , address=104.36.82.7:4909 , piece_id=GBAB3QNPQNYRZJZC4NCVYWAIITLQRLYIVO6IW5QJ2LA4TPYNVSVA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.236967972 (2h21m57.017901341s) [noise=true , address=104.36.82.15:4257 , piece_id=4EEEMA3HQLS6WI25OF7NWWJH5M2TMVEEUFL7TDUZ2KXDQIJKVIMQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 159.223.143.39:51340->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.236995526 (2h21m57.017928895s) [noise=true , address=104.36.82.4:4801 , piece_id=IEXEXGC2OSYX6TFQ7ELEYSBAG4T2UPH4RJ2VC5QD47RYQNFORFZA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.237035393 (2h21m57.01796877s) [noise=true , address=77.90.31.21:4878 , piece_id=NOVSKD6JTPKZV2HNFOMLPST2XBWK4VHANKEDOBEA23HFPHYQJM3Q, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:49578->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.237062045 (2h21m57.017995421s) [seg_pos={0 0}, upload=2169]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:13:41.260380631 (2h21m57.041313999s) [upload=2169]: Done uploading segments... Jul 27 00:13:41.260406109 (2h21m57.041339474s) [upload=2169]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 596.559873ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X] duration: 30.025659628s FAIL (2025-07-27 00:13:41 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ddress=104.36.82.6:4875 , piece_id=GHIW37HRIVD6GT3SFE5NG37KJTJPMTXIPZ3DFBD2FZIPDZTWMHBA, seg_pos={0 0}, upload=2158]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 159.223.143.39:51714->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:40.438474259 (2h20m56.219407638s) [seg_pos={0 0}, upload=2158]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:12:40.438560347 (2h20m56.21949373s) [upload=2158]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:12:40.438566838 (2h20m56.219500227s) [upload=2158]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:12:40.509422921 (2h20m56.290356315s) [upload=2158]: Done uploading segments... Jul 27 00:12:40.509464315 (2h20m56.290397697s) [upload=2158]: Upload finished. err: <nil> Jul 27 00:13:11.277010948 (2h21m27.057944326s) [upload=2169]: Starting upload Jul 27 00:13:11.277116731 (2h21m27.058050112s) [upload=2169]: Done starting upload Jul 27 00:13:11.277197399 (2h21m27.058130775s) [upload=2169]: Uploading segments... Jul 27 00:13:11.277594524 (2h21m27.058527912s) [upload=2169]: Got next segment. Inline: false Jul 27 00:13:11.277672198 (2h21m27.058605573s) [upload=2169]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:13:11.277679743 (2h21m27.05861311s) [upload=2169]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:13:11.313276916 (2h21m27.094210302s) [seg_pos={0 0}, upload=2169]: Begin upload segment... Jul 27 00:13:11.313357802 (2h21m27.09429119s) [seg_pos={0 0}, upload=2169]: Done begin upload segment. Jul 27 00:13:11.313377828 (2h21m27.094311218s) [noise=true , address=45.91.116.5:4299 , piece_id=SVY5VQZOXDDBRYBR6F3GZVTN2U2P3V5XEVIKGMTPFSKHHTTGHCBA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313399798 (2h21m27.094333185s) [noise=true , address=45.91.116.14:4191 , piece_id=DKXQ24HLMYCNABL2WZXL5UTWA5KEOS6YTOJBOQOS5OXIHXYNZBWQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313517824 (2h21m27.09445121s) [noise=true , address=104.36.82.7:4909 , piece_id=GBAB3QNPQNYRZJZC4NCVYWAIITLQRLYIVO6IW5QJ2LA4TPYNVSVA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313934720 (2h21m27.094868105s) [noise=true , address=77.90.31.19:4842 , piece_id=WQ2GNUFNPH7G4UQECIOPQQ4K5HOUWSZKS35SE4PTK7Y26AOPA2EA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.313964599 (2h21m27.094897984s) [noise=true , address=77.90.31.16:4733 , piece_id=I6UAOCUNR5KK35UKYYWY47CIZD7GZMRHAYS5QLFUZWHSED4NXILQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.314326344 (2h21m27.095259726s) [noise=true , address=104.36.82.15:4257 , piece_id=4EEEMA3HQLS6WI25OF7NWWJH5M2TMVEEUFL7TDUZ2KXDQIJKVIMQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.314429730 (2h21m27.095363118s) [noise=true , address=45.91.116.8:4045 , piece_id=V3IOJG5S7SZD725R7S22YOOF7VEKO2LA44EW3WTF7DUZE67OACHQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.314882565 (2h21m27.095815949s) [noise=true , address=77.90.31.15:4699 , piece_id=5JDLEXIEI3GZMRGMMGQ7FHHNFB3ZBJ32SHNZ2VNNFROJNBZYALCQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.315377830 (2h21m27.096311218s) [noise=true , address=45.91.116.6:4974 , piece_id=CRWDOM4F7CPPXU3VD3HATLB2LGO5Q2ABZPGMR5TTKHVPX55VVROQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.315696091 (2h21m27.096629479s) [noise=true , address=45.91.116.12:4226 , piece_id=NBX4ULNVNJ6VZ3X3K5P57QNS52K2QSEA7LOXBM73SCHPEW344QQA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.316218266 (2h21m27.097151651s) [noise=true , address=45.91.116.11:4119 , piece_id=N3VPTFS3Q2MZSKT53MQLP7VR6R37V6TZ5LHDLJPP4ITAKQFD6MTQ, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.316632666 (2h21m27.097566057s) [noise=true , address=104.36.82.4:4801 , piece_id=IEXEXGC2OSYX6TFQ7ELEYSBAG4T2UPH4RJ2VC5QD47RYQNFORFZA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.316810856 (2h21m27.097744245s) [noise=true , address=45.91.116.4:4939 , piece_id=GX4K5CDRJJRZ6UMBVK2GGYIHUJLDQVYB4D63PCCKPEM42DGKOVTA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.317278579 (2h21m27.09821197s) [noise=true , address=45.91.116.9:4263 , piece_id=3AYY7VIOP44PCPN5IBNRYPL7CGOP6SLBHCZBNIK7HA7DBVQW4R4Q, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.317748179 (2h21m27.098681568s) [noise=true , address=77.90.31.21:4878 , piece_id=NOVSKD6JTPKZV2HNFOMLPST2XBWK4VHANKEDOBEA23HFPHYQJM3Q, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.318152255 (2h21m27.099085649s) [noise=true , address=45.91.116.4:4940 , piece_id=ORVKXYTD3TSCNJ4ZY5KGF3Y6DOCXII2C4M3NRZ3XRNVHZEM4WZWA, seg_pos={0 0}, upload=2169]: Uploading piece... Jul 27 00:13:11.427694395 (2h21m27.208627777s) [upload=2169]: Next returned nil segment Jul 27 00:13:11.427714857 (2h21m27.208648243s) [upload=2169]: Waiting for error group managing segments... Jul 27 00:13:11.446118534 (2h21m27.227051975s) [noise=true , address=45.91.116.11:4119 , piece_id=N3VPTFS3Q2MZSKT53MQLP7VR6R37V6TZ5LHDLJPP4ITAKQFD6MTQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.446797332 (2h21m27.227730717s) [noise=true , address=77.90.31.16:4733 , piece_id=I6UAOCUNR5KK35UKYYWY47CIZD7GZMRHAYS5QLFUZWHSED4NXILQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.447424597 (2h21m27.22835796s) [noise=true , address=45.91.116.6:4974 , piece_id=CRWDOM4F7CPPXU3VD3HATLB2LGO5Q2ABZPGMR5TTKHVPX55VVROQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.447949057 (2h21m27.22888242s) [noise=true , address=77.90.31.15:4699 , piece_id=5JDLEXIEI3GZMRGMMGQ7FHHNFB3ZBJ32SHNZ2VNNFROJNBZYALCQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.448485189 (2h21m27.229418576s) [noise=true , address=45.91.116.4:4939 , piece_id=GX4K5CDRJJRZ6UMBVK2GGYIHUJLDQVYB4D63PCCKPEM42DGKOVTA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.449018483 (2h21m27.229951844s) [noise=true , address=45.91.116.9:4263 , piece_id=3AYY7VIOP44PCPN5IBNRYPL7CGOP6SLBHCZBNIK7HA7DBVQW4R4Q, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.449709382 (2h21m27.23064276s) [noise=true , address=45.91.116.8:4045 , piece_id=V3IOJG5S7SZD725R7S22YOOF7VEKO2LA44EW3WTF7DUZE67OACHQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.449742304 (2h21m27.230675678s) [noise=true , address=45.91.116.5:4299 , piece_id=SVY5VQZOXDDBRYBR6F3GZVTN2U2P3V5XEVIKGMTPFSKHHTTGHCBA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.450341755 (2h21m27.231275132s) [noise=true , address=45.91.116.4:4940 , piece_id=ORVKXYTD3TSCNJ4ZY5KGF3Y6DOCXII2C4M3NRZ3XRNVHZEM4WZWA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.450449996 (2h21m27.231383369s) [noise=true , address=45.91.116.12:4226 , piece_id=NBX4ULNVNJ6VZ3X3K5P57QNS52K2QSEA7LOXBM73SCHPEW344QQA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:11.692726076 (2h21m27.473659458s) [noise=true , address=45.91.116.14:4191 , piece_id=DKXQ24HLMYCNABL2WZXL5UTWA5KEOS6YTOJBOQOS5OXIHXYNZBWQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:12.789383367 (2h21m28.570316735s) [noise=true , address=77.90.31.19:4842 , piece_id=WQ2GNUFNPH7G4UQECIOPQQ4K5HOUWSZKS35SE4PTK7Y26AOPA2EA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: <nil> Jul 27 00:13:41.236943285 (2h21m57.017876661s) [noise=true , address=104.36.82.7:4909 , piece_id=GBAB3QNPQNYRZJZC4NCVYWAIITLQRLYIVO6IW5QJ2LA4TPYNVSVA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.236967972 (2h21m57.017901341s) [noise=true , address=104.36.82.15:4257 , piece_id=4EEEMA3HQLS6WI25OF7NWWJH5M2TMVEEUFL7TDUZ2KXDQIJKVIMQ, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 159.223.143.39:51340->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.236995526 (2h21m57.017928895s) [noise=true , address=104.36.82.4:4801 , piece_id=IEXEXGC2OSYX6TFQ7ELEYSBAG4T2UPH4RJ2VC5QD47RYQNFORFZA, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.237035393 (2h21m57.01796877s) [noise=true , address=77.90.31.21:4878 , piece_id=NOVSKD6JTPKZV2HNFOMLPST2XBWK4VHANKEDOBEA23HFPHYQJM3Q, seg_pos={0 0}, upload=2169]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:49578->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:41.237062045 (2h21m57.017995421s) [seg_pos={0 0}, upload=2169]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:13:41.260380631 (2h21m57.041313999s) [upload=2169]: Done uploading segments... Jul 27 00:13:41.260406109 (2h21m57.041339474s) [upload=2169]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.025688641s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.X....XX..............X.X........X...XX.....................] duration: 23.371782548s FAIL (2025-07-27 00:13:27 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.371801055s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..] duration: 1.914269926s FAIL (2025-07-27 00:13:12 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 1.914313048s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.037236912s FAIL (2025-07-27 00:13:11 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== CU3HOGU6TMIIKY7SNONKN2AKTKZPYT6INIAFWW7263LQ, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.474779599 (2h20m27.255712978s) [seg_pos={0 0}, upload=2155]: Done begin upload segment. Jul 27 00:12:11.474839535 (2h20m27.255772932s) [noise=true , address=104.36.82.47:4539 , piece_id=HPCS5VETT23V2Q2XXHJ2IV3FFHVAY3RM7E5HZ4S3CDE5FLMUZ5KQ, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.475380867 (2h20m27.256314255s) [noise=true , address=104.36.82.50:4042 , piece_id=IRDFI5V3RVD6LFTH6GYMTXPK6O6FGFRBX2WNHD5HUITEPTRLJXOA, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.475780199 (2h20m27.256713588s) [noise=true , address=104.36.82.46:4496 , piece_id=5XADA2PAJA56CACQDFB3SR6B4MHKUU5M5BTJZCRMOJN6IX5Y2SMA, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.476239578 (2h20m27.257172963s) [noise=true , address=79.127.205.245:4236 , piece_id=V6YBOCATK7SDTM3S5HAQ2MCOYH3KEROBUL63BAQGD6QW36R2EO7Q, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.476626885 (2h20m27.257560272s) [noise=true , address=79.127.220.98:4422 , piece_id=2CMSLO3DR2UCF2EZUOLXLFIVEIQ3FA52GKJJEPI2RFUK2ECSIQPQ, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.476998417 (2h20m27.257931803s) [noise=true , address=79.127.205.246:4273 , piece_id=TOXLEMIZKT64PLVG3R43TCTN2D5VII2ELXVPCIOGY3QPTGC2VQQA, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.477370215 (2h20m27.2583036s) [noise=true , address=64.49.13.138:4483 , piece_id=4AFIYCUAOANY2OZPSUP2S4VBPV2QSJHT3B6PPL75SQOQPMAOCZ5A, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.477739767 (2h20m27.258673152s) [noise=true , address=77.90.31.13:4658 , piece_id=4MZSVUZMV6LWRTLHO6ENREQ4V2LHMZS7HSDVAJMBDIROHDNICU4Q, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.478112353 (2h20m27.259045739s) [noise=true , address=79.127.205.234:4618 , piece_id=DB62765ADIO7FHV33NTRA5OGLYOYFSA4NYL5H6YXAIBBKAMLQUQA, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.478466946 (2h20m27.259400332s) [noise=true , address=104.36.82.53:4129 , piece_id=PSXK2NHZNE7JCK5EZKYZ6HW65AZWF4N6I7JAUNIIYROTSJT5G3YA, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.478825049 (2h20m27.259758436s) [noise=true , address=79.127.219.38:4015 , piece_id=SO2WLO42CR43XLPEPTYN2X4DPF4X7PAWMY5WVFDTBZDGUKWHJ6IQ, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.479220255 (2h20m27.260153656s) [noise=true , address=104.36.82.54:4165 , piece_id=SMB6HVMSJQAI642P4HVCJJDK4EHKR3VLCERLVBVC3FTEAFRR5Q6A, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.479580033 (2h20m27.260513418s) [noise=true , address=104.36.82.13:4193 , piece_id=AHGIBTBMMANFITTACM7QMPY7ODDTVKV436YWS7AYAOQG2GJH4I5Q, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.479929192 (2h20m27.260862577s) [noise=true , address=77.90.31.21:4891 , piece_id=TRGPVOVUB7TWMWVIFLEBAI4FQH2BCPEAMKIWJ6I5HDLWFEZE6AXQ, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.480317609 (2h20m27.261250994s) [noise=true , address=64.49.13.138:4482 , piece_id=OWAIK3H7ATCXD3AWGCMVAN4KRML75H22M2YMWAAB7CPW4ZPDXOWA, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.480670267 (2h20m27.261603657s) [noise=true , address=104.36.82.52:4116 , piece_id=KP4GODZ2LBFIOFHWWVFVCXYVY6GOXTFT7LF3NWEFBQ2O2MSSIP6A, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.487416063 (2h20m27.268349449s) [noise=true , address=77.90.31.18:4823 , piece_id=D625KEDIYDI5AKQC4IJKEIQFWHNANDDX5KNOMBTANCLLP4LNUWHA, seg_pos={0 0}, upload=2155]: Uploading piece... Jul 27 00:12:11.561344810 (2h20m27.342278187s) [upload=2155]: Next returned nil segment Jul 27 00:12:11.561363418 (2h20m27.342296794s) [upload=2155]: Waiting for error group managing segments... Jul 27 00:12:11.564909881 (2h20m27.345843249s) [noise=true , address=79.127.219.38:4015 , piece_id=SO2WLO42CR43XLPEPTYN2X4DPF4X7PAWMY5WVFDTBZDGUKWHJ6IQ, seg_pos={0 0}, upload=2155]: Done uploading piece. err: <nil> Jul 27 00:12:12.069193818 (2h20m27.850127181s) [noise=true , address=79.127.205.233:4652 , piece_id=IJUYLAWRCU3HOGU6TMIIKY7SNONKN2AKTKZPYT6INIAFWW7263LQ, seg_pos={0 0}, upload=2155]: Done uploading piece. err: <nil> Jul 27 00:12:12.119366539 (2h20m27.900299923s) [noise=true , address=79.127.205.246:4273 , piece_id=TOXLEMIZKT64PLVG3R43TCTN2D5VII2ELXVPCIOGY3QPTGC2VQQA, seg_pos={0 0}, upload=2155]: Done uploading piece. err: <nil> Jul 27 00:12:12.121822371 (2h20m27.902755735s) [noise=true , address=79.127.205.245:4236 , piece_id=V6YBOCATK7SDTM3S5HAQ2MCOYH3KEROBUL63BAQGD6QW36R2EO7Q, seg_pos={0 0}, upload=2155]: Done uploading piece. err: <nil> Jul 27 00:12:12.140666801 (2h20m27.921600169s) [noise=true , address=79.127.205.234:4618 , piece_id=DB62765ADIO7FHV33NTRA5OGLYOYFSA4NYL5H6YXAIBBKAMLQUQA, seg_pos={0 0}, upload=2155]: Done uploading piece. err: <nil> Jul 27 00:12:12.715547694 (2h20m28.49648108s) [noise=true , address=77.90.31.13:4658 , piece_id=4MZSVUZMV6LWRTLHO6ENREQ4V2LHMZS7HSDVAJMBDIROHDNICU4Q, seg_pos={0 0}, upload=2155]: Done uploading piece. err: <nil> Jul 27 00:12:14.641219316 (2h20m30.4221527s) [noise=true , address=79.127.220.98:4422 , piece_id=2CMSLO3DR2UCF2EZUOLXLFIVEIQ3FA52GKJJEPI2RFUK2ECSIQPQ, seg_pos={0 0}, upload=2155]: Done uploading piece. err: <nil> Jul 27 00:12:21.456468361 (2h20m37.237401727s) [noise=true , address=64.49.13.138:4483 , piece_id=4AFIYCUAOANY2OZPSUP2S4VBPV2QSJHT3B6PPL75SQOQPMAOCZ5A, seg_pos={0 0}, upload=2155]: Done uploading piece. err: <nil> Jul 27 00:12:22.866703724 (2h20m38.647637086s) [noise=true , address=64.49.13.138:4482 , piece_id=OWAIK3H7ATCXD3AWGCMVAN4KRML75H22M2YMWAAB7CPW4ZPDXOWA, seg_pos={0 0}, upload=2155]: Done uploading piece. err: <nil> Jul 27 00:13:11.396553355 (2h21m27.177486746s) [noise=true , address=104.36.82.13:4193 , piece_id=AHGIBTBMMANFITTACM7QMPY7ODDTVKV436YWS7AYAOQG2GJH4I5Q, seg_pos={0 0}, upload=2155]: Done uploading piece. err: ecclient: upload failed (node:15XNLgptmXprcfkVMJsuzY2WDyHiZDpLetiBpmjXhb4c9q7yMd, address:104.36.82.13:4193): protocol: rpc: dial tcp 104.36.82.13:4193: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:11.396683683 (2h21m27.177617072s) [noise=true , address=104.36.82.53:4129 , piece_id=PSXK2NHZNE7JCK5EZKYZ6HW65AZWF4N6I7JAUNIIYROTSJT5G3YA, seg_pos={0 0}, upload=2155]: Done uploading piece. err: ecclient: upload failed (node:1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, address:104.36.82.53:4129): protocol: rpc: dial tcp 104.36.82.53:4129: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:11.396752474 (2h21m27.177685853s) [noise=true , address=104.36.82.54:4165 , piece_id=SMB6HVMSJQAI642P4HVCJJDK4EHKR3VLCERLVBVC3FTEAFRR5Q6A, seg_pos={0 0}, upload=2155]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: rpc: dial tcp 104.36.82.54:4165: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:11.396819753 (2h21m27.177753133s) [noise=true , address=77.90.31.21:4891 , piece_id=TRGPVOVUB7TWMWVIFLEBAI4FQH2BCPEAMKIWJ6I5HDLWFEZE6AXQ, seg_pos={0 0}, upload=2155]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: rpc: dial tcp 77.90.31.21:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:11.397226423 (2h21m27.178159809s) [noise=true , address=77.90.31.18:4823 , piece_id=D625KEDIYDI5AKQC4IJKEIQFWHNANDDX5KNOMBTANCLLP4LNUWHA, seg_pos={0 0}, upload=2155]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:13:11.397353794 (2h21m27.178287173s) [noise=true , address=104.36.82.46:4496 , piece_id=5XADA2PAJA56CACQDFB3SR6B4MHKUU5M5BTJZCRMOJN6IX5Y2SMA, seg_pos={0 0}, upload=2155]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:13:11.397444729 (2h21m27.178378109s) [noise=true , address=104.36.82.47:4539 , piece_id=HPCS5VETT23V2Q2XXHJ2IV3FFHVAY3RM7E5HZ4S3CDE5FLMUZ5KQ, seg_pos={0 0}, upload=2155]: Done uploading piece. err: ecclient: upload failed (node:126g75XgAyFYcgnaibseqoGiz7cK6z3n6gPMcu9TrdjuZUYEHnZ, address:104.36.82.47:4539): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:13:11.397919985 (2h21m27.178853365s) [noise=true , address=104.36.82.50:4042 , piece_id=IRDFI5V3RVD6LFTH6GYMTXPK6O6FGFRBX2WNHD5HUITEPTRLJXOA, seg_pos={0 0}, upload=2155]: Done uploading piece. err: ecclient: upload failed (node:1LbRrZsqp1TR354jZQLstDmWAUBVBMyid8AuAbTfQj7N89Mnte, address:104.36.82.50:4042): protocol: rpc: dial tcp 104.36.82.50:4042: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:11.397990342 (2h21m27.17892372s) [noise=true , address=104.36.82.52:4116 , piece_id=KP4GODZ2LBFIOFHWWVFVCXYVY6GOXTFT7LF3NWEFBQ2O2MSSIP6A, seg_pos={0 0}, upload=2155]: Done uploading piece. err: ecclient: upload failed (node:12vKCXgWTAsipTvUK9XA3sL6DznFA8BqZc6WCN84PBuLhKCTT9m, address:104.36.82.52:4116): protocol: write tcp 159.223.143.39:37602->104.36.82.52:4116: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:11.398015537 (2h21m27.178948918s) [seg_pos={0 0}, upload=2155]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:13:11.429641576 (2h21m27.210574947s) [upload=2155]: Done uploading segments... Jul 27 00:13:11.429669188 (2h21m27.210602576s) [upload=2155]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.037262861s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..X....XX..............X.X........X...XX....................] duration: 46.84525729s FAIL (2025-07-27 00:13:04 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 46.845300214s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...XX.....X.XXX...............X..X......X.XX....X...X.......] duration: 551.263327ms FAIL (2025-07-27 00:12:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 551.285881ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.] duration: 1.935421076s FAIL (2025-07-27 00:12:40 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 314542506 (2h19m48.095475886s) [upload=2145]: Done starting upload Jul 27 00:11:32.314833287 (2h19m48.095766663s) [upload=2145]: Uploading segments... Jul 27 00:11:32.314893632 (2h19m48.095827017s) [upload=2145]: Got next segment. Inline: false Jul 27 00:11:32.314934607 (2h19m48.095867986s) [upload=2145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:11:32.314940536 (2h19m48.095873952s) [upload=2145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:11:32.358989266 (2h19m48.139922631s) [seg_pos={0 0}, upload=2145]: Begin upload segment... Jul 27 00:11:32.359080795 (2h19m48.14001416s) [seg_pos={0 0}, upload=2145]: Done begin upload segment. Jul 27 00:11:32.359112107 (2h19m48.140045483s) [noise=true , address=45.91.116.14:4189 , piece_id=CSFAV42VOI5QRMMKSV7QDVO2TKFCM6Z7IYQAGOM5PNNQXCBCBRFA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.359131009 (2h19m48.140064388s) [noise=true , address=104.36.82.9:4007 , piece_id=MAMJSEK3LWTZFU4SVWB423H4CHY4PKSPWXMVPXHIJXBFCVJISFTA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.359557235 (2h19m48.14049062s) [noise=true , address=77.90.31.14:4662 , piece_id=P36FVJKX2LB2KOZZ2DNDLPWTKWFZH25DL4I5ECU4H46TEFHYODUA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.359672407 (2h19m48.140605785s) [noise=true , address=104.36.82.4:4802 , piece_id=U2UGSTTSZIJD4KXJHP74UQDAGUTLJJR3OUAUM6QKU4NXZ6XBTLJA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.359868527 (2h19m48.140801904s) [noise=true , address=104.36.82.6:4875 , piece_id=5Y7PJY2MRASMGJ6FNOLJNX7SJBWYHEKHIZUUEHFYD245JQQJQWOQ, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.360315728 (2h19m48.141249109s) [noise=true , address=104.36.82.3:4767 , piece_id=OXF6W345J543R3ECXVOP2B3TZ5RJCR4TD2LOD4FA6BWXQGONGL2Q, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.360940298 (2h19m48.141873678s) [noise=true , address=104.36.82.4:4801 , piece_id=CWHLGRRJWKGWVCNHEH3KV724JZQHMB4N26KVCHSHLKUL6VXMN5CA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.363171520 (2h19m48.144104901s) [noise=true , address=45.91.116.12:4225 , piece_id=JJMGQYWNJNZPPHXAIOCCB5ZSCCL2C6F5Q43OIFUMFYXCEG5RDCRQ, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.363609540 (2h19m48.144542915s) [noise=true , address=104.36.82.11:4113 , piece_id=6ZNSFERPUCAOXXICJKRIL62AB5TQIYEP7VZ5MUFP36XDLXR5OMBA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.363934473 (2h19m48.144867872s) [noise=true , address=77.90.31.12:4590 , piece_id=NS54QU2R3THW2NDGL67OYIYBSEFFAEPBGMJE3C7W5QHMXE7NDGRQ, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.364418032 (2h19m48.145351412s) [noise=true , address=104.36.82.13:4186 , piece_id=S3C5O57CE4IMVVQVWAYGDNVBJHSW4WRMYSAPMIXE6SL3PZ3LHYIQ, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.364847636 (2h19m48.145781026s) [noise=true , address=77.90.31.18:4805 , piece_id=RUFOLEREXO44OUQYZK7ELTSM7NPQEPPRXU65BQEA3Z2ODAZ7QJFA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.365229070 (2h19m48.146162462s) [noise=true , address=77.90.31.19:4842 , piece_id=IWKPAHR4I4YCPWGSNJORN3ZZ76SR2ROC5NNYUHFJZGTO3PV6XLZA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.365580258 (2h19m48.146513638s) [noise=true , address=45.91.116.13:4155 , piece_id=Z4A4S3PANYPRSQNKTJVWET2OFINLFHNG7QEU2I47REPNSIPUNY7A, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.365888285 (2h19m48.146821663s) [noise=true , address=104.36.82.5:4839 , piece_id=3NGQM7LNMA36RB6MI6T2EK3TEOP2MV5THOKGNQCZSJMHSQC2O4ZQ, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.366216950 (2h19m48.147150328s) [noise=true , address=77.90.31.16:4735 , piece_id=DT3Z2CODEPADAJM643RFK67YOTS4E65KJUXNN4UVJX6ZV7WZM32A, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.435644764 (2h19m48.216578142s) [upload=2145]: Next returned nil segment Jul 27 00:11:32.435666271 (2h19m48.216599637s) [upload=2145]: Waiting for error group managing segments... Jul 27 00:11:32.440416261 (2h19m48.221349639s) [noise=true , address=45.91.116.13:4155 , piece_id=Z4A4S3PANYPRSQNKTJVWET2OFINLFHNG7QEU2I47REPNSIPUNY7A, seg_pos={0 0}, upload=2145]: Done uploading piece. err: <nil> Jul 27 00:11:32.440451732 (2h19m48.221385102s) [noise=true , address=45.91.116.12:4225 , piece_id=JJMGQYWNJNZPPHXAIOCCB5ZSCCL2C6F5Q43OIFUMFYXCEG5RDCRQ, seg_pos={0 0}, upload=2145]: Done uploading piece. err: <nil> Jul 27 00:11:32.440950422 (2h19m48.221883793s) [noise=true , address=77.90.31.14:4662 , piece_id=P36FVJKX2LB2KOZZ2DNDLPWTKWFZH25DL4I5ECU4H46TEFHYODUA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: <nil> Jul 27 00:11:32.440987609 (2h19m48.221920979s) [noise=true , address=77.90.31.16:4735 , piece_id=DT3Z2CODEPADAJM643RFK67YOTS4E65KJUXNN4UVJX6ZV7WZM32A, seg_pos={0 0}, upload=2145]: Done uploading piece. err: <nil> Jul 27 00:11:32.848839884 (2h19m48.629773253s) [noise=true , address=45.91.116.14:4189 , piece_id=CSFAV42VOI5QRMMKSV7QDVO2TKFCM6Z7IYQAGOM5PNNQXCBCBRFA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: <nil> Jul 27 00:11:33.836650446 (2h19m49.617583807s) [noise=true , address=77.90.31.19:4842 , piece_id=IWKPAHR4I4YCPWGSNJORN3ZZ76SR2ROC5NNYUHFJZGTO3PV6XLZA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: <nil> Jul 27 00:11:33.837905779 (2h19m49.618839149s) [noise=true , address=77.90.31.12:4590 , piece_id=NS54QU2R3THW2NDGL67OYIYBSEFFAEPBGMJE3C7W5QHMXE7NDGRQ, seg_pos={0 0}, upload=2145]: Done uploading piece. err: <nil> Jul 27 00:12:02.285759366 (2h20m18.066692752s) [noise=true , address=104.36.82.11:4113 , piece_id=6ZNSFERPUCAOXXICJKRIL62AB5TQIYEP7VZ5MUFP36XDLXR5OMBA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.285860565 (2h20m18.066793953s) [noise=true , address=104.36.82.5:4839 , piece_id=3NGQM7LNMA36RB6MI6T2EK3TEOP2MV5THOKGNQCZSJMHSQC2O4ZQ, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.285936009 (2h20m18.066869378s) [noise=true , address=104.36.82.9:4007 , piece_id=MAMJSEK3LWTZFU4SVWB423H4CHY4PKSPWXMVPXHIJXBFCVJISFTA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.285991545 (2h20m18.066924913s) [noise=true , address=104.36.82.4:4802 , piece_id=U2UGSTTSZIJD4KXJHP74UQDAGUTLJJR3OUAUM6QKU4NXZ6XBTLJA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.286048764 (2h20m18.066982132s) [noise=true , address=104.36.82.6:4875 , piece_id=5Y7PJY2MRASMGJ6FNOLJNX7SJBWYHEKHIZUUEHFYD245JQQJQWOQ, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.286105406 (2h20m18.067038773s) [noise=true , address=104.36.82.3:4767 , piece_id=OXF6W345J543R3ECXVOP2B3TZ5RJCR4TD2LOD4FA6BWXQGONGL2Q, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.286160478 (2h20m18.067093844s) [noise=true , address=104.36.82.4:4801 , piece_id=CWHLGRRJWKGWVCNHEH3KV724JZQHMB4N26KVCHSHLKUL6VXMN5CA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.286228850 (2h20m18.067162216s) [noise=true , address=104.36.82.13:4186 , piece_id=S3C5O57CE4IMVVQVWAYGDNVBJHSW4WRMYSAPMIXE6SL3PZ3LHYIQ, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.286292018 (2h20m18.067225381s) [noise=true , address=77.90.31.18:4805 , piece_id=RUFOLEREXO44OUQYZK7ELTSM7NPQEPPRXU65BQEA3Z2ODAZ7QJFA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 159.223.143.39:34508->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.286313912 (2h20m18.067247312s) [seg_pos={0 0}, upload=2145]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 27 00:12:02.311015245 (2h20m18.091948638s) [upload=2145]: Done uploading segments... Jul 27 00:12:02.311076840 (2h20m18.092010226s) [upload=2145]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 1.935456927s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X...XX.....X.XXX...............X..X......X.XX....X...X......] duration: 29.59103493s FAIL (2025-07-27 00:12:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.591080299s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 48 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X.] duration: 1.859897059s FAIL (2025-07-27 00:12:27 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 1.859928615s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X] duration: 30.029744931s FAIL (2025-07-27 00:12:02 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 314542506 (2h19m48.095475886s) [upload=2145]: Done starting upload Jul 27 00:11:32.314833287 (2h19m48.095766663s) [upload=2145]: Uploading segments... Jul 27 00:11:32.314893632 (2h19m48.095827017s) [upload=2145]: Got next segment. Inline: false Jul 27 00:11:32.314934607 (2h19m48.095867986s) [upload=2145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:11:32.314940536 (2h19m48.095873952s) [upload=2145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:11:32.358989266 (2h19m48.139922631s) [seg_pos={0 0}, upload=2145]: Begin upload segment... Jul 27 00:11:32.359080795 (2h19m48.14001416s) [seg_pos={0 0}, upload=2145]: Done begin upload segment. Jul 27 00:11:32.359112107 (2h19m48.140045483s) [noise=true , address=45.91.116.14:4189 , piece_id=CSFAV42VOI5QRMMKSV7QDVO2TKFCM6Z7IYQAGOM5PNNQXCBCBRFA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.359131009 (2h19m48.140064388s) [noise=true , address=104.36.82.9:4007 , piece_id=MAMJSEK3LWTZFU4SVWB423H4CHY4PKSPWXMVPXHIJXBFCVJISFTA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.359557235 (2h19m48.14049062s) [noise=true , address=77.90.31.14:4662 , piece_id=P36FVJKX2LB2KOZZ2DNDLPWTKWFZH25DL4I5ECU4H46TEFHYODUA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.359672407 (2h19m48.140605785s) [noise=true , address=104.36.82.4:4802 , piece_id=U2UGSTTSZIJD4KXJHP74UQDAGUTLJJR3OUAUM6QKU4NXZ6XBTLJA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.359868527 (2h19m48.140801904s) [noise=true , address=104.36.82.6:4875 , piece_id=5Y7PJY2MRASMGJ6FNOLJNX7SJBWYHEKHIZUUEHFYD245JQQJQWOQ, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.360315728 (2h19m48.141249109s) [noise=true , address=104.36.82.3:4767 , piece_id=OXF6W345J543R3ECXVOP2B3TZ5RJCR4TD2LOD4FA6BWXQGONGL2Q, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.360940298 (2h19m48.141873678s) [noise=true , address=104.36.82.4:4801 , piece_id=CWHLGRRJWKGWVCNHEH3KV724JZQHMB4N26KVCHSHLKUL6VXMN5CA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.363171520 (2h19m48.144104901s) [noise=true , address=45.91.116.12:4225 , piece_id=JJMGQYWNJNZPPHXAIOCCB5ZSCCL2C6F5Q43OIFUMFYXCEG5RDCRQ, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.363609540 (2h19m48.144542915s) [noise=true , address=104.36.82.11:4113 , piece_id=6ZNSFERPUCAOXXICJKRIL62AB5TQIYEP7VZ5MUFP36XDLXR5OMBA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.363934473 (2h19m48.144867872s) [noise=true , address=77.90.31.12:4590 , piece_id=NS54QU2R3THW2NDGL67OYIYBSEFFAEPBGMJE3C7W5QHMXE7NDGRQ, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.364418032 (2h19m48.145351412s) [noise=true , address=104.36.82.13:4186 , piece_id=S3C5O57CE4IMVVQVWAYGDNVBJHSW4WRMYSAPMIXE6SL3PZ3LHYIQ, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.364847636 (2h19m48.145781026s) [noise=true , address=77.90.31.18:4805 , piece_id=RUFOLEREXO44OUQYZK7ELTSM7NPQEPPRXU65BQEA3Z2ODAZ7QJFA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.365229070 (2h19m48.146162462s) [noise=true , address=77.90.31.19:4842 , piece_id=IWKPAHR4I4YCPWGSNJORN3ZZ76SR2ROC5NNYUHFJZGTO3PV6XLZA, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.365580258 (2h19m48.146513638s) [noise=true , address=45.91.116.13:4155 , piece_id=Z4A4S3PANYPRSQNKTJVWET2OFINLFHNG7QEU2I47REPNSIPUNY7A, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.365888285 (2h19m48.146821663s) [noise=true , address=104.36.82.5:4839 , piece_id=3NGQM7LNMA36RB6MI6T2EK3TEOP2MV5THOKGNQCZSJMHSQC2O4ZQ, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.366216950 (2h19m48.147150328s) [noise=true , address=77.90.31.16:4735 , piece_id=DT3Z2CODEPADAJM643RFK67YOTS4E65KJUXNN4UVJX6ZV7WZM32A, seg_pos={0 0}, upload=2145]: Uploading piece... Jul 27 00:11:32.435644764 (2h19m48.216578142s) [upload=2145]: Next returned nil segment Jul 27 00:11:32.435666271 (2h19m48.216599637s) [upload=2145]: Waiting for error group managing segments... Jul 27 00:11:32.440416261 (2h19m48.221349639s) [noise=true , address=45.91.116.13:4155 , piece_id=Z4A4S3PANYPRSQNKTJVWET2OFINLFHNG7QEU2I47REPNSIPUNY7A, seg_pos={0 0}, upload=2145]: Done uploading piece. err: <nil> Jul 27 00:11:32.440451732 (2h19m48.221385102s) [noise=true , address=45.91.116.12:4225 , piece_id=JJMGQYWNJNZPPHXAIOCCB5ZSCCL2C6F5Q43OIFUMFYXCEG5RDCRQ, seg_pos={0 0}, upload=2145]: Done uploading piece. err: <nil> Jul 27 00:11:32.440950422 (2h19m48.221883793s) [noise=true , address=77.90.31.14:4662 , piece_id=P36FVJKX2LB2KOZZ2DNDLPWTKWFZH25DL4I5ECU4H46TEFHYODUA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: <nil> Jul 27 00:11:32.440987609 (2h19m48.221920979s) [noise=true , address=77.90.31.16:4735 , piece_id=DT3Z2CODEPADAJM643RFK67YOTS4E65KJUXNN4UVJX6ZV7WZM32A, seg_pos={0 0}, upload=2145]: Done uploading piece. err: <nil> Jul 27 00:11:32.848839884 (2h19m48.629773253s) [noise=true , address=45.91.116.14:4189 , piece_id=CSFAV42VOI5QRMMKSV7QDVO2TKFCM6Z7IYQAGOM5PNNQXCBCBRFA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: <nil> Jul 27 00:11:33.836650446 (2h19m49.617583807s) [noise=true , address=77.90.31.19:4842 , piece_id=IWKPAHR4I4YCPWGSNJORN3ZZ76SR2ROC5NNYUHFJZGTO3PV6XLZA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: <nil> Jul 27 00:11:33.837905779 (2h19m49.618839149s) [noise=true , address=77.90.31.12:4590 , piece_id=NS54QU2R3THW2NDGL67OYIYBSEFFAEPBGMJE3C7W5QHMXE7NDGRQ, seg_pos={0 0}, upload=2145]: Done uploading piece. err: <nil> Jul 27 00:12:02.285759366 (2h20m18.066692752s) [noise=true , address=104.36.82.11:4113 , piece_id=6ZNSFERPUCAOXXICJKRIL62AB5TQIYEP7VZ5MUFP36XDLXR5OMBA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.285860565 (2h20m18.066793953s) [noise=true , address=104.36.82.5:4839 , piece_id=3NGQM7LNMA36RB6MI6T2EK3TEOP2MV5THOKGNQCZSJMHSQC2O4ZQ, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.285936009 (2h20m18.066869378s) [noise=true , address=104.36.82.9:4007 , piece_id=MAMJSEK3LWTZFU4SVWB423H4CHY4PKSPWXMVPXHIJXBFCVJISFTA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.285991545 (2h20m18.066924913s) [noise=true , address=104.36.82.4:4802 , piece_id=U2UGSTTSZIJD4KXJHP74UQDAGUTLJJR3OUAUM6QKU4NXZ6XBTLJA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.286048764 (2h20m18.066982132s) [noise=true , address=104.36.82.6:4875 , piece_id=5Y7PJY2MRASMGJ6FNOLJNX7SJBWYHEKHIZUUEHFYD245JQQJQWOQ, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.286105406 (2h20m18.067038773s) [noise=true , address=104.36.82.3:4767 , piece_id=OXF6W345J543R3ECXVOP2B3TZ5RJCR4TD2LOD4FA6BWXQGONGL2Q, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.286160478 (2h20m18.067093844s) [noise=true , address=104.36.82.4:4801 , piece_id=CWHLGRRJWKGWVCNHEH3KV724JZQHMB4N26KVCHSHLKUL6VXMN5CA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.286228850 (2h20m18.067162216s) [noise=true , address=104.36.82.13:4186 , piece_id=S3C5O57CE4IMVVQVWAYGDNVBJHSW4WRMYSAPMIXE6SL3PZ3LHYIQ, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.286292018 (2h20m18.067225381s) [noise=true , address=77.90.31.18:4805 , piece_id=RUFOLEREXO44OUQYZK7ELTSM7NPQEPPRXU65BQEA3Z2ODAZ7QJFA, seg_pos={0 0}, upload=2145]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 159.223.143.39:34508->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:02.286313912 (2h20m18.067247312s) [seg_pos={0 0}, upload=2145]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 27 00:12:02.311015245 (2h20m18.091948638s) [upload=2145]: Done uploading segments... Jul 27 00:12:02.311076840 (2h20m18.092010226s) [upload=2145]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.029802338s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X] duration: 30.032137158s FAIL (2025-07-27 00:12:01 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.032167336s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...X....XX..............X.X........X...XX...................] duration: 27.890976862s FAIL (2025-07-27 00:12:01 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.890994989s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.025346953s FAIL (2025-07-27 00:11:58 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 7A, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.881067654 (2h19m14.662001045s) [noise=true , address=104.36.82.40:4959 , piece_id=O2CPROZZVXGEJ6YKX2LCBSNC2P6Y4KMPKTACKT2PZUVLA3B3VDBA, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.881452462 (2h19m14.662385852s) [noise=true , address=104.36.82.18:4399 , piece_id=GNDN3YWCUOYRWNZUUMVEMKKM5H7SQ7IQ5JWKARZ7OP3KSLHDHBHA, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.881819908 (2h19m14.662753296s) [noise=true , address=104.36.82.42:4805 , piece_id=PSMYKJWVGDAFPEVQMOGZOJFR4ZYT7K7MCCBPEFP6E3BTS5CWHB3A, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.882190364 (2h19m14.663123756s) [noise=true , address=104.36.82.46:4514 , piece_id=HNYGBOI56NAFJROB3EWRJOYUA4XK5GO6VNKCQLBWFFJKM6ZCHRLA, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.882556447 (2h19m14.663489836s) [noise=true , address=104.36.82.56:4249 , piece_id=2ZWUAGX76VFWDJQPCSMJJRTMZXDIQJJ4XW4Y4SRGAKL4HVWI3HQA, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.882973579 (2h19m14.66390696s) [noise=true , address=104.36.82.47:4535 , piece_id=PR7H75DZ6HZ5I6SILCR4EE6K57P6OYJKMZD44OYZVA2HFF6A5PFA, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.883384581 (2h19m14.664317958s) [noise=true , address=104.36.82.55:4215 , piece_id=P6ZNQ2F2RPRYNPGO3ROV6VCNW3UN6H7GG3M2L2DRLMRZVKY4NMGQ, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.883873128 (2h19m14.664806515s) [noise=true , address=79.127.205.233:4639 , piece_id=FOZNY7FXI2QCJRR7OZU26AHY2JX2W5MLMWETM2QZSNX7SDTSBU7Q, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.884495674 (2h19m14.665429051s) [noise=true , address=79.127.219.38:4015 , piece_id=XL2ITS4UWY5ODJUWRRC2BXBV6S6YNECZE2FZPXN2JH2SNMEXR3GQ, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.884972886 (2h19m14.665906273s) [noise=true , address=104.36.82.54:4171 , piece_id=UHBGTYY72VGGPTBRDO7ZQWVYCF7ZZNCUVF6YHGZK7TDQRU3JLRBA, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.885408826 (2h19m14.666342216s) [noise=true , address=77.90.31.10:4528 , piece_id=5FUET73FWANJX6M55UBWQFTUUW7DR33Y525QV5UIKQK2ZDIFT3AA, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.885896278 (2h19m14.666829671s) [noise=true , address=79.127.201.215:4350 , piece_id=2E4HNJHTK4E3OQ7DF4SWRNUUYTMOEBT5WCIQ664AFOR52JDL6RMQ, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.886837619 (2h19m14.667770997s) [noise=true , address=64.49.13.140:4514 , piece_id=BPBPL5LQO6NG4ONEXX72GOLAGLGSIJ4PXLBOVBOK7XNEWSYX7X7A, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.887260539 (2h19m14.66819392s) [noise=true , address=104.36.82.52:4105 , piece_id=2ME3LRSUZEW2W6NZSZQNSWDF6N6F2ISULK6DGBCHXCGMV2ZV4MQQ, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.887630196 (2h19m14.668563577s) [noise=true , address=77.90.31.16:4739 , piece_id=TY2XBOMIVHR2FGKSPXBUF3INBH3YGCR2YQKR4PL5J2V72Y4SPQVQ, seg_pos={0 0}, upload=2135]: Uploading piece... Jul 27 00:10:58.980653446 (2h19m14.761586835s) [upload=2135]: Next returned nil segment Jul 27 00:10:58.980678763 (2h19m14.761612152s) [upload=2135]: Waiting for error group managing segments... Jul 27 00:10:58.997545538 (2h19m14.778478919s) [noise=true , address=77.90.31.10:4528 , piece_id=5FUET73FWANJX6M55UBWQFTUUW7DR33Y525QV5UIKQK2ZDIFT3AA, seg_pos={0 0}, upload=2135]: Done uploading piece. err: <nil> Jul 27 00:10:59.017538436 (2h19m14.7984718s) [noise=true , address=77.90.31.16:4739 , piece_id=TY2XBOMIVHR2FGKSPXBUF3INBH3YGCR2YQKR4PL5J2V72Y4SPQVQ, seg_pos={0 0}, upload=2135]: Done uploading piece. err: <nil> Jul 27 00:10:59.073488736 (2h19m14.854422106s) [noise=true , address=79.127.219.38:4015 , piece_id=XL2ITS4UWY5ODJUWRRC2BXBV6S6YNECZE2FZPXN2JH2SNMEXR3GQ, seg_pos={0 0}, upload=2135]: Done uploading piece. err: <nil> Jul 27 00:10:59.258126114 (2h19m15.039059503s) [noise=true , address=64.49.13.140:4514 , piece_id=BPBPL5LQO6NG4ONEXX72GOLAGLGSIJ4PXLBOVBOK7XNEWSYX7X7A, seg_pos={0 0}, upload=2135]: Done uploading piece. err: ecclient: upload failed (node:1WsbDX2Zy8FYnEQtUsYFHSeDHj1DkkGyWBPaxzmReWFzaTejHn, address:64.49.13.140:4514): protocol: storage node overloaded, request limit: 2000 Jul 27 00:10:59.525879494 (2h19m15.306812857s) [noise=true , address=79.127.205.233:4639 , piece_id=FOZNY7FXI2QCJRR7OZU26AHY2JX2W5MLMWETM2QZSNX7SDTSBU7Q, seg_pos={0 0}, upload=2135]: Done uploading piece. err: <nil> Jul 27 00:11:01.690030080 (2h19m17.470963465s) [noise=true , address=79.127.220.101:4048 , piece_id=SWQK3DG2OGZZN3Z4JNKGFAWAOPHHK74WM5JMGGJBN6FRWGU2HOHA, seg_pos={0 0}, upload=2135]: Done uploading piece. err: <nil> Jul 27 00:11:03.773632824 (2h19m19.554566206s) [noise=true , address=77.90.31.19:4871 , piece_id=MDPGZ5MUGDD5H2XNE6PV7X73R5TQOXBDU6RSM4CZTECRF44LB67A, seg_pos={0 0}, upload=2135]: Done uploading piece. err: <nil> Jul 27 00:11:20.458380627 (2h19m36.239313989s) [noise=true , address=79.127.201.215:4350 , piece_id=2E4HNJHTK4E3OQ7DF4SWRNUUYTMOEBT5WCIQ664AFOR52JDL6RMQ, seg_pos={0 0}, upload=2135]: Done uploading piece. err: <nil> Jul 27 00:11:58.811887908 (2h20m14.592821273s) [noise=true , address=104.36.82.18:4399 , piece_id=GNDN3YWCUOYRWNZUUMVEMKKM5H7SQ7IQ5JWKARZ7OP3KSLHDHBHA, seg_pos={0 0}, upload=2135]: Done uploading piece. err: ecclient: upload failed (node:1QPwXaFGpwMLXMiSnkE1eycqZkUSnDYMj9sFYZ2BPrPbenw9uK, address:104.36.82.18:4399): protocol: rpc: dial tcp 104.36.82.18:4399: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:58.811962735 (2h20m14.592896101s) [noise=true , address=104.36.82.55:4215 , piece_id=P6ZNQ2F2RPRYNPGO3ROV6VCNW3UN6H7GG3M2L2DRLMRZVKY4NMGQ, seg_pos={0 0}, upload=2135]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:58.812055287 (2h20m14.592988678s) [noise=true , address=104.36.82.54:4171 , piece_id=UHBGTYY72VGGPTBRDO7ZQWVYCF7ZZNCUVF6YHGZK7TDQRU3JLRBA, seg_pos={0 0}, upload=2135]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: rpc: dial tcp 104.36.82.54:4171: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:58.812318752 (2h20m14.593252132s) [noise=true , address=104.36.82.40:4959 , piece_id=O2CPROZZVXGEJ6YKX2LCBSNC2P6Y4KMPKTACKT2PZUVLA3B3VDBA, seg_pos={0 0}, upload=2135]: Done uploading piece. err: ecclient: upload failed (node:1NCN7sTw12KVDpD8zLjUxQki2MdMuDJyEnPSirG8ni5L2xi7aa, address:104.36.82.40:4959): protocol: rpc: dial tcp 104.36.82.40:4959: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:58.812410763 (2h20m14.59334414s) [noise=true , address=104.36.82.42:4805 , piece_id=PSMYKJWVGDAFPEVQMOGZOJFR4ZYT7K7MCCBPEFP6E3BTS5CWHB3A, seg_pos={0 0}, upload=2135]: Done uploading piece. err: ecclient: upload failed (node:1nh6Kih617STw2M8iHVjs4xP1HjjckJHyUxQ5z6ZFRhWwG6RXi, address:104.36.82.42:4805): protocol: rpc: dial tcp 104.36.82.42:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:58.812478404 (2h20m14.593411779s) [noise=true , address=104.36.82.52:4105 , piece_id=2ME3LRSUZEW2W6NZSZQNSWDF6N6F2ISULK6DGBCHXCGMV2ZV4MQQ, seg_pos={0 0}, upload=2135]: Done uploading piece. err: ecclient: upload failed (node:1TVtMQNVV2kERp2aZwmS8E9JUiua1RCyFpZgFACRqH1Nfscw4C, address:104.36.82.52:4105): protocol: rpc: dial tcp 104.36.82.52:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:58.812487111 (2h20m14.593420472s) [noise=true , address=104.36.82.47:4535 , piece_id=PR7H75DZ6HZ5I6SILCR4EE6K57P6OYJKMZD44OYZVA2HFF6A5PFA, seg_pos={0 0}, upload=2135]: Done uploading piece. err: ecclient: upload failed (node:12KFwCcqBCTyeBBQYzg6SFG4YDzBm2H9s4D2JRvnC31VAGvpLhu, address:104.36.82.47:4535): protocol: rpc: dial tcp 104.36.82.47:4535: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:58.812533012 (2h20m14.593466388s) [noise=true , address=104.36.82.46:4514 , piece_id=HNYGBOI56NAFJROB3EWRJOYUA4XK5GO6VNKCQLBWFFJKM6ZCHRLA, seg_pos={0 0}, upload=2135]: Done uploading piece. err: ecclient: upload failed (node:1iJgoxnaKnevARfWPAkd6ygyKXHZeVH8BvJPH13zbev6mGw2Bk, address:104.36.82.46:4514): protocol: rpc: dial tcp 104.36.82.46:4514: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:58.812553071 (2h20m14.593486437s) [noise=true , address=104.36.82.56:4249 , piece_id=2ZWUAGX76VFWDJQPCSMJJRTMZXDIQJJ4XW4Y4SRGAKL4HVWI3HQA, seg_pos={0 0}, upload=2135]: Done uploading piece. err: ecclient: upload failed (node:1LuzmiCefHA9kX3HHWHqviHfG4Ts2t5DYu5nfJRWquaQ8nQrzx, address:104.36.82.56:4249): protocol: write tcp 159.223.143.39:43292->104.36.82.56:4249: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:58.812576055 (2h20m14.593509439s) [noise=true , address=104.36.82.11:4146 , piece_id=F26GRGJBPE72Y65P6GHG4KSAHOQGTCKRJ4VGMIFHULZWJUEYSJ7A, seg_pos={0 0}, upload=2135]: Done uploading piece. err: ecclient: upload failed (node:1Sv7eCy62QTiVT1c7mxzJSTAnrCsDoRrxqxfL6govEFRJRQJ6V, address:104.36.82.11:4146): protocol: rpc: dial tcp 104.36.82.11:4146: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:58.812597995 (2h20m14.593531381s) [seg_pos={0 0}, upload=2135]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 27 00:11:58.835232957 (2h20m14.616166336s) [upload=2135]: Done uploading segments... Jul 27 00:11:58.835258332 (2h20m14.616191717s) [upload=2135]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.025362491s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XX...XX.....X.XXX...............X..X......X.XX....X...X.....] duration: 643.75865ms FAIL (2025-07-27 00:11:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 643.853753ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [X...X....XX..............X.X........X...XX..................] duration: 36.047153233s FAIL (2025-07-27 00:11:33 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 36.047193027s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.029669534s FAIL (2025-07-27 00:10:58 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 00:09:58.870609219 (2h18m14.651542604s) [noise=true , address=156.146.43.225:4449 , piece_id=ED56GBHDKQKYJYJUXCEWJXZ2SE42ZBNOVE2ESHN57KQJMA7RNTXA, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.870621625 (2h18m14.651555001s) [noise=true , address=104.36.82.46:4497 , piece_id=G7QQZJ5BY7VEOGYGNBDJRG5FB7MEL66HKNHMXUQ2IKU7FV53FYFQ, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.871073095 (2h18m14.652006476s) [noise=true , address=79.127.205.233:4645 , piece_id=XNZHWR6TWYKD7PU6M7UZYYYN2ZV6N4NFH7O7DIABHT3J547CNK4A, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.871085543 (2h18m14.652018924s) [noise=true , address=77.90.31.18:4823 , piece_id=JXXOOKSXTFU52RFTWR6N4HJGP2EDABRLY54DIEKLCGJ73T2UB5AA, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.871342139 (2h18m14.652275521s) [noise=true , address=104.36.82.47:4529 , piece_id=633WRSG6WGI7TFOEAODZEVW2IA3ECPTXBBBLF6IOJTEATZRHQKWA, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.871466604 (2h18m14.652399982s) [noise=true , address=104.36.82.55:4214 , piece_id=WONYGNS4GCOIVA4H5MQEU4RUPX6CR5WF73VJSHECWLYFQWHLOB2A, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.871587034 (2h18m14.652520415s) [noise=true , address=104.36.82.53:4140 , piece_id=2RN6O4BKSKG446J5EJ7MRM5XLGN33BLR3UQEWWY6BCWI2QSZ5JWQ, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.871827370 (2h18m14.652760731s) [noise=true , address=104.36.82.56:4251 , piece_id=SFRKCK6KDDQHVX2FLUMGWRYCGSGL5DIG43RJYMBGAUDB6S3SYPEA, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.871841213 (2h18m14.652774591s) [noise=true , address=79.127.205.246:4285 , piece_id=WGIZ4OG7LDIZL2SOMQAUYTWC333UPRE5UDGV5M2PSYVTOOOWCL3A, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.872348907 (2h18m14.653282288s) [noise=true , address=77.90.31.10:4536 , piece_id=4KQKXHZUXDUOAPA2UWGMBWLAZSQE225IEWLTZLFY6DHPKVAUGIFA, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.872738965 (2h18m14.653672343s) [noise=true , address=79.127.201.215:4348 , piece_id=S4YE4KIUI7Y4UH7WQVS7PTFAA6FT7BAOETDUCZ64LA6E7OTXEM2A, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.873122358 (2h18m14.65405574s) [noise=true , address=45.91.116.9:4277 , piece_id=YOOP7ICN6VJM33ROHABESV7F45QWWIVZ2C5UCXQZBF2OSPUYKUGA, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.873787435 (2h18m14.6547208s) [noise=true , address=79.127.219.38:4015 , piece_id=V5S4T4YYBFIH5XOX3OFU3TPCQSBC2NHQA6ZVOX3GP7PVRBU6YKXQ, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.874096515 (2h18m14.655029901s) [noise=true , address=64.49.13.26:4953 , piece_id=SGQUJKG46EFTE63625ZPQ5ALK2SP6RSQX7YH2SO4OYL5K5IO5JEQ, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.874995372 (2h18m14.655928757s) [noise=true , address=79.127.205.234:4607 , piece_id=357BKCQLOJI25YBAGTUMBGF6EB2734SZDHZTBVQCYTSLILZ5NIGA, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.876171222 (2h18m14.657104609s) [noise=true , address=104.36.82.51:4062 , piece_id=FURTWHCQJJ5BTGZUNXSR3LZIXKIDZX6AO4VOUGM6GO55VIJMBHQQ, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.877288285 (2h18m14.65822167s) [noise=true , address=79.127.205.245:4247 , piece_id=OTD7OZKXNXQT4AAGZ4KTTKQWGBMAYLV6WQKNZ32I42K5DH7QQNPA, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.878143783 (2h18m14.659077173s) [noise=true , address=45.91.116.14:4194 , piece_id=VMSG5JXP4QHLTBIPEFBZCYLILAW76LF2U5ESVRV5MLMJ7RZWGQBA, seg_pos={0 0}, upload=2117]: Uploading piece... Jul 27 00:09:58.954651744 (2h18m14.735585121s) [upload=2117]: Next returned nil segment Jul 27 00:09:58.954674071 (2h18m14.735607454s) [upload=2117]: Waiting for error group managing segments... Jul 27 00:09:58.981385914 (2h18m14.762319282s) [noise=true , address=79.127.219.38:4015 , piece_id=V5S4T4YYBFIH5XOX3OFU3TPCQSBC2NHQA6ZVOX3GP7PVRBU6YKXQ, seg_pos={0 0}, upload=2117]: Done uploading piece. err: <nil> Jul 27 00:09:58.995594996 (2h18m14.776528375s) [noise=true , address=77.90.31.10:4536 , piece_id=4KQKXHZUXDUOAPA2UWGMBWLAZSQE225IEWLTZLFY6DHPKVAUGIFA, seg_pos={0 0}, upload=2117]: Done uploading piece. err: <nil> Jul 27 00:09:59.444828594 (2h18m15.22576197s) [noise=true , address=79.127.205.233:4645 , piece_id=XNZHWR6TWYKD7PU6M7UZYYYN2ZV6N4NFH7O7DIABHT3J547CNK4A, seg_pos={0 0}, upload=2117]: Done uploading piece. err: <nil> Jul 27 00:09:59.473968428 (2h18m15.254901804s) [noise=true , address=79.127.205.246:4285 , piece_id=WGIZ4OG7LDIZL2SOMQAUYTWC333UPRE5UDGV5M2PSYVTOOOWCL3A, seg_pos={0 0}, upload=2117]: Done uploading piece. err: <nil> Jul 27 00:09:59.485903479 (2h18m15.266836857s) [noise=true , address=79.127.205.245:4247 , piece_id=OTD7OZKXNXQT4AAGZ4KTTKQWGBMAYLV6WQKNZ32I42K5DH7QQNPA, seg_pos={0 0}, upload=2117]: Done uploading piece. err: <nil> Jul 27 00:09:59.767063697 (2h18m15.547997077s) [noise=true , address=79.127.205.234:4607 , piece_id=357BKCQLOJI25YBAGTUMBGF6EB2734SZDHZTBVQCYTSLILZ5NIGA, seg_pos={0 0}, upload=2117]: Done uploading piece. err: <nil> Jul 27 00:10:03.561863986 (2h18m19.342797355s) [noise=true , address=156.146.43.225:4449 , piece_id=ED56GBHDKQKYJYJUXCEWJXZ2SE42ZBNOVE2ESHN57KQJMA7RNTXA, seg_pos={0 0}, upload=2117]: Done uploading piece. err: <nil> Jul 27 00:10:04.426823907 (2h18m20.207757282s) [noise=true , address=45.91.116.14:4194 , piece_id=VMSG5JXP4QHLTBIPEFBZCYLILAW76LF2U5ESVRV5MLMJ7RZWGQBA, seg_pos={0 0}, upload=2117]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: storage node overloaded, request limit: 2000 Jul 27 00:10:05.779230864 (2h18m21.560164233s) [noise=true , address=64.49.13.26:4953 , piece_id=SGQUJKG46EFTE63625ZPQ5ALK2SP6RSQX7YH2SO4OYL5K5IO5JEQ, seg_pos={0 0}, upload=2117]: Done uploading piece. err: <nil> Jul 27 00:10:08.236128125 (2h18m24.017061513s) [noise=true , address=45.91.116.9:4277 , piece_id=YOOP7ICN6VJM33ROHABESV7F45QWWIVZ2C5UCXQZBF2OSPUYKUGA, seg_pos={0 0}, upload=2117]: Done uploading piece. err: <nil> Jul 27 00:10:35.516470487 (2h18m51.297403865s) [noise=true , address=79.127.201.215:4348 , piece_id=S4YE4KIUI7Y4UH7WQVS7PTFAA6FT7BAOETDUCZ64LA6E7OTXEM2A, seg_pos={0 0}, upload=2117]: Done uploading piece. err: <nil> Jul 27 00:10:58.708431972 (2h19m14.489365337s) [noise=true , address=104.36.82.53:4140 , piece_id=2RN6O4BKSKG446J5EJ7MRM5XLGN33BLR3UQEWWY6BCWI2QSZ5JWQ, seg_pos={0 0}, upload=2117]: Done uploading piece. err: ecclient: upload failed (node:12Wv1U4bCkzg9Ywx4EKRkWG8tMffjwiVBEZ6tHr8scNHfdcWyrL, address:104.36.82.53:4140): protocol: rpc: dial tcp 104.36.82.53:4140: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:10:58.708543339 (2h19m14.489476717s) [noise=true , address=77.90.31.18:4823 , piece_id=JXXOOKSXTFU52RFTWR6N4HJGP2EDABRLY54DIEKLCGJ73T2UB5AA, seg_pos={0 0}, upload=2117]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: rpc: dial tcp 77.90.31.18:4823: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:10:58.708621455 (2h19m14.489554842s) [noise=true , address=104.36.82.51:4062 , piece_id=FURTWHCQJJ5BTGZUNXSR3LZIXKIDZX6AO4VOUGM6GO55VIJMBHQQ, seg_pos={0 0}, upload=2117]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: write tcp 159.223.143.39:37192->104.36.82.51:4062: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:10:58.708683851 (2h19m14.48961722s) [noise=true , address=104.36.82.55:4214 , piece_id=WONYGNS4GCOIVA4H5MQEU4RUPX6CR5WF73VJSHECWLYFQWHLOB2A, seg_pos={0 0}, upload=2117]: Done uploading piece. err: ecclient: upload failed (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi, address:104.36.82.55:4214): protocol: write tcp 159.223.143.39:44640->104.36.82.55:4214: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:10:58.708733161 (2h19m14.489666524s) [noise=true , address=104.36.82.46:4497 , piece_id=G7QQZJ5BY7VEOGYGNBDJRG5FB7MEL66HKNHMXUQ2IKU7FV53FYFQ, seg_pos={0 0}, upload=2117]: Done uploading piece. err: ecclient: upload failed (node:12Zg4PSYBuLxFvF9oYiuvn65JNbjT5RdagcVAQLp2izvjPe1NEc, address:104.36.82.46:4497): protocol: write tcp 159.223.143.39:51100->104.36.82.46:4497: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:10:58.708755188 (2h19m14.489688552s) [noise=true , address=104.36.82.47:4529 , piece_id=633WRSG6WGI7TFOEAODZEVW2IA3ECPTXBBBLF6IOJTEATZRHQKWA, seg_pos={0 0}, upload=2117]: Done uploading piece. err: ecclient: upload failed (node:1m7FrQEps7d8enRYk4v17UxcDs8o8L2CFcVEyB3zR4EniXrYeH, address:104.36.82.47:4529): protocol: write tcp 159.223.143.39:45894->104.36.82.47:4529: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:10:58.708784893 (2h19m14.489718261s) [noise=true , address=104.36.82.56:4251 , piece_id=SFRKCK6KDDQHVX2FLUMGWRYCGSGL5DIG43RJYMBGAUDB6S3SYPEA, seg_pos={0 0}, upload=2117]: Done uploading piece. err: ecclient: upload failed (node:12vvYKd1KXrz8cYXRJSL9KXiYQtY95aB2GFPGFi9c9Nj1Raj6J4, address:104.36.82.56:4251): protocol: rpc: dial tcp 104.36.82.56:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:10:58.708841198 (2h19m14.489774585s) [seg_pos={0 0}, upload=2117]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:10:58.735372401 (2h19m14.516305779s) [upload=2117]: Done uploading segments... Jul 27 00:10:58.735424354 (2h19m14.516357734s) [upload=2117]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.029690923s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.X...X....XX..............X.X........X...XX.................] duration: 49.239530897s FAIL (2025-07-27 00:10:57 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 49.239609603s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XXX...XX.....X.XXX...............X..X......X.XX....X...X....] duration: 703.311818ms FAIL (2025-07-27 00:10:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 703.343065ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....] duration: 2.63219441s FAIL (2025-07-27 00:10:45 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.632250455s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..] duration: 17.699187622s FAIL (2025-07-27 00:10:37 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== _id=OTZ6FQJRXFS3GWQIC2OZKBS4GR56LPUISPEDHWWQP7EONXTK4RZQ, seg_pos={0 0}, upload=2083]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:08:19.740998689 (2h16m35.521932116s) [noise=true , address=77.90.31.15:4699 , piece_id=LFCQEJG76VOFKXE2ZCGRXL7W2VNJL4TU4JYZU57F4DXJ55M7AYTQ, seg_pos={0 0}, upload=2083]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:08:19.741048351 (2h16m35.52198174s) [seg_pos={0 0}, upload=2083]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:08:19.764313990 (2h16m35.545247374s) [upload=2083]: Done uploading segments... Jul 27 00:08:19.764346822 (2h16m35.545280207s) [upload=2083]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:08:45.635298021 (2h17m1.4162315s) [upload=2097]: Starting upload Jul 27 00:08:45.635390125 (2h17m1.416323529s) [upload=2097]: Done starting upload Jul 27 00:08:45.635618718 (2h17m1.416552096s) [upload=2097]: Uploading segments... Jul 27 00:08:45.635780082 (2h17m1.416713462s) [upload=2097]: Got next segment. Inline: false Jul 27 00:08:45.635855437 (2h17m1.416788814s) [upload=2097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:08:45.635863070 (2h17m1.416796461s) [upload=2097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:08:45.666562989 (2h17m1.447496374s) [seg_pos={0 0}, upload=2097]: Begin upload segment... Jul 27 00:08:45.666633979 (2h17m1.447567347s) [seg_pos={0 0}, upload=2097]: Done begin upload segment. Jul 27 00:08:45.666674450 (2h17m1.447607829s) [noise=true , address=104.36.82.11:4115 , piece_id=FUAGHJHK7O64SCY2DIQ4ZH242XIELALGZFKHZBNDHADD6MNTSGXA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.666685892 (2h17m1.447619261s) [noise=true , address=45.91.116.5:4299 , piece_id=FLF5N2NDPNYOCYCNIGNWVY4WB2LURWFKXNAA7GNY6FOBSW3J64HA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.666754668 (2h17m1.447688044s) [noise=true , address=45.91.116.6:4975 , piece_id=DXOFPBUKYFW6GKR3V2UIYDIYLEIU3WCXL2ZIKUDNII6ABHSVDO6A, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667156517 (2h17m1.448089903s) [noise=true , address=45.91.116.4:4937 , piece_id=2KHWB574OKRBSHJYEM7VACSJNJDCDFXL6KLXBCXX7YU5M2PBPTAQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667216106 (2h17m1.448149484s) [noise=true , address=77.90.31.10:4518 , piece_id=7HCOYWWIANMJTHEJFMF6R67UALQWEHKDVGBD2XTHXNR4R2CGYSGQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667535834 (2h17m1.448469218s) [noise=true , address=45.91.116.8:4045 , piece_id=ZDGLNB566Q33PRSBA5CAYNCN76QWPJHMQMA2KGYIR3RRZR724V3Q, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667719777 (2h17m1.448653163s) [noise=true , address=45.91.116.10:4081 , piece_id=JX2VJ33Y7BXQSQAQ64RTKRQBCTI7VO6E4LQQGCSBZ7K5QVW2CO3Q, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667900825 (2h17m1.448834217s) [noise=true , address=77.90.31.14:4662 , piece_id=H6K6AETN3ZXM2SX7OYCTZEV5D5OXBNVEO2RZRGINGKSU7EFKCMVA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.668432972 (2h17m1.449366357s) [noise=true , address=104.36.82.13:4185 , piece_id=VUMNOYB6N5HVW7C3EG3IRYNF4K5GWYRAAFINFHAOLLDBUCAVO65Q, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.668922889 (2h17m1.449856273s) [noise=true , address=45.91.116.12:4225 , piece_id=UEJV3Q5Z74OJMIXHHDDXU7JX7SDZ2AMTVQAMBOT5ERJ4TJAXAXTA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.671253980 (2h17m1.452187358s) [noise=true , address=45.91.116.11:4117 , piece_id=X2AJPZ5QO4JM3E2S4NZ2RNZXUH5XC22NJ7YG63GJUV57T3GICPXA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.671603059 (2h17m1.452536438s) [noise=true , address=77.90.31.13:4628 , piece_id=H65YSRTR7DUDGFN7FMD3L64ENKLHADKZKDTU3XN6D64IHQUVEATQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.671890649 (2h17m1.452824036s) [noise=true , address=77.90.31.17:4772 , piece_id=6J7FD26JEJZUQLUXFIZIPFQ5WEJTLRMVEKCYLE744W32HK6RKDVQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.673442036 (2h17m1.454375416s) [noise=true , address=77.90.31.15:4698 , piece_id=GHPGFOMGZFLOAGWDOTZQWXISMJKUBO63TQ2CIOV653OUQYEUX5IQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.673811853 (2h17m1.454745223s) [noise=true , address=77.90.31.21:4879 , piece_id=4HXLD5QEEHXEQGYACGH6CNMH5VJGJP3434ICNU2WUGC44WQHG54A, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.686682455 (2h17m1.467615818s) [noise=true , address=104.36.82.5:4839 , piece_id=O3OAVQXCCBSNJBNGXSU2SZO5FZVRHUVBCY74KE2344PJY5OUGRRQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.771440822 (2h17m1.552374209s) [upload=2097]: Next returned nil segment Jul 27 00:08:45.771463691 (2h17m1.55239708s) [upload=2097]: Waiting for error group managing segments... Jul 27 00:08:45.775342955 (2h17m1.556276318s) [noise=true , address=45.91.116.4:4937 , piece_id=2KHWB574OKRBSHJYEM7VACSJNJDCDFXL6KLXBCXX7YU5M2PBPTAQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.776475944 (2h17m1.557409313s) [noise=true , address=45.91.116.5:4299 , piece_id=FLF5N2NDPNYOCYCNIGNWVY4WB2LURWFKXNAA7GNY6FOBSW3J64HA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.777015261 (2h17m1.557948628s) [noise=true , address=45.91.116.11:4117 , piece_id=X2AJPZ5QO4JM3E2S4NZ2RNZXUH5XC22NJ7YG63GJUV57T3GICPXA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.777591233 (2h17m1.55852462s) [noise=true , address=45.91.116.10:4081 , piece_id=JX2VJ33Y7BXQSQAQ64RTKRQBCTI7VO6E4LQQGCSBZ7K5QVW2CO3Q, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.778018122 (2h17m1.558951495s) [noise=true , address=45.91.116.8:4045 , piece_id=ZDGLNB566Q33PRSBA5CAYNCN76QWPJHMQMA2KGYIR3RRZR724V3Q, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.778418119 (2h17m1.559351488s) [noise=true , address=77.90.31.14:4662 , piece_id=H6K6AETN3ZXM2SX7OYCTZEV5D5OXBNVEO2RZRGINGKSU7EFKCMVA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.778929404 (2h17m1.559862779s) [noise=true , address=45.91.116.6:4975 , piece_id=DXOFPBUKYFW6GKR3V2UIYDIYLEIU3WCXL2ZIKUDNII6ABHSVDO6A, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.779537556 (2h17m1.560470931s) [noise=true , address=77.90.31.17:4772 , piece_id=6J7FD26JEJZUQLUXFIZIPFQ5WEJTLRMVEKCYLE744W32HK6RKDVQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.779812214 (2h17m1.560745582s) [noise=true , address=45.91.116.12:4225 , piece_id=UEJV3Q5Z74OJMIXHHDDXU7JX7SDZ2AMTVQAMBOT5ERJ4TJAXAXTA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:46.684702430 (2h17m2.465635798s) [noise=true , address=77.90.31.21:4879 , piece_id=4HXLD5QEEHXEQGYACGH6CNMH5VJGJP3434ICNU2WUGC44WQHG54A, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:47.073910770 (2h17m2.854844136s) [noise=true , address=77.90.31.10:4518 , piece_id=7HCOYWWIANMJTHEJFMF6R67UALQWEHKDVGBD2XTHXNR4R2CGYSGQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:47.166542112 (2h17m2.947475475s) [noise=true , address=77.90.31.13:4628 , piece_id=H65YSRTR7DUDGFN7FMD3L64ENKLHADKZKDTU3XN6D64IHQUVEATQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:09:15.606542690 (2h17m31.387476056s) [noise=true , address=104.36.82.5:4839 , piece_id=O3OAVQXCCBSNJBNGXSU2SZO5FZVRHUVBCY74KE2344PJY5OUGRRQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:15.606697150 (2h17m31.387630526s) [noise=true , address=77.90.31.15:4698 , piece_id=GHPGFOMGZFLOAGWDOTZQWXISMJKUBO63TQ2CIOV653OUQYEUX5IQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:15.606735183 (2h17m31.387668552s) [noise=true , address=104.36.82.13:4185 , piece_id=VUMNOYB6N5HVW7C3EG3IRYNF4K5GWYRAAFINFHAOLLDBUCAVO65Q, seg_pos={0 0}, upload=2097]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 159.223.143.39:37514->104.36.82.13:4185: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:15.606756595 (2h17m31.387689958s) [noise=true , address=104.36.82.11:4115 , piece_id=FUAGHJHK7O64SCY2DIQ4ZH242XIELALGZFKHZBNDHADD6MNTSGXA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:15.606778960 (2h17m31.387712338s) [seg_pos={0 0}, upload=2097]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:09:15.635367371 (2h17m31.416300738s) [upload=2097]: Done uploading segments... Jul 27 00:09:15.635393090 (2h17m31.416326454s) [upload=2097]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 17.699222667s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.XXX...XX.....X.XXX...............X..X......X.XX....X...X...] duration: 690.33045ms FAIL (2025-07-27 00:10:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 690.44241ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 50 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX...] duration: 1.927472761s FAIL (2025-07-27 00:10:13 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 1.927584006s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [..X...X....XX..............X.X........X...XX................] duration: 28.575118033s FAIL (2025-07-27 00:10:07 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.57514105s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.027240879s FAIL (2025-07-27 00:09:58 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 49.13.138:4483 , piece_id=KKN7VFWOFSS6UKGMVP4O5SP34MM2KZDOCRQH7QFGD4YFQIE6YKDA, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.693844258 (2h17m14.474777655s) [noise=true , address=77.90.31.15:4730 , piece_id=WVSI5HFXAUSKHLRAN4CASOGODTD4X73RDCOS4GRLRTNVUREUBY7A, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.694569395 (2h17m14.475502783s) [noise=true , address=77.90.31.15:4714 , piece_id=3HZWM765U3XAABGE3NTW5JYHDVQNRXOIKUJU2R67FFYFZIWPMYKA, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.695217237 (2h17m14.47615063s) [noise=true , address=45.91.116.14:4206 , piece_id=H5BFYVDSTQKN2ZPRNOPFPS4EEUO6UY5H425OE4346QCHBBSL53CQ, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.695781767 (2h17m14.476715145s) [noise=true , address=104.36.82.54:4169 , piece_id=PKI7VE3YP6M6NMCW4IA5UU7RMTEL4WLATOGAHAJAOYOPTWTVJJJQ, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.696331308 (2h17m14.477264688s) [noise=true , address=77.90.31.19:4856 , piece_id=VQQMI4YAFAPADAXDQWUUWVGMRSYEYBV2LCKXFBBOAEKBKSZ6TNWQ, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.696780964 (2h17m14.477714348s) [noise=true , address=104.36.82.52:4107 , piece_id=C7RYDOFWHJSWRXUKO56UXWXTLZ5KNXKQFTAHIOQ5OIP34ESWS7PQ, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.697383991 (2h17m14.478317386s) [noise=true , address=79.127.205.245:4227 , piece_id=R33XV7QC723PQWKM5PR6QTVYBDBGUHVVMXQKM5ROSZ77W5J4JSAA, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.697999609 (2h17m14.478932999s) [noise=true , address=104.36.82.56:4234 , piece_id=JX7MT6BQCEATXGPPHXGGGZTWPMS4MXURILRAH4TUBA6QJIPGCHNQ, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.698573705 (2h17m14.47950709s) [noise=true , address=79.127.219.48:4074 , piece_id=RCKEXJFHLJTFCEK5CTIFWIPTT4OG7CCAHWJ5FZWVP2BMV6MF2ADA, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.699167283 (2h17m14.480100673s) [noise=true , address=64.49.13.138:4482 , piece_id=A64AWGLBYK4RANPOSQDPEFFL5TNJ6HWRPIEPDYY4IT3DZM5J53ZQ, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.699714793 (2h17m14.480648181s) [noise=true , address=104.36.82.55:4202 , piece_id=DPQYMMBYQZ4DM5IQLMZCIKF4HCGPAWRIVMDO3VPQ6SR4ZX37JIUQ, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.705705924 (2h17m14.486639308s) [noise=true , address=79.127.205.246:4281 , piece_id=WJ7SXHUVGEE36O7IKYTEUID3MBQ4BT6WEGSR5GOZ24WCOBF3FSPA, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.706439516 (2h17m14.487372902s) [noise=true , address=104.36.82.46:4513 , piece_id=OIAQWPQOYAYG3Q2BFKVCRYZPY5ACJ5M3RXMU6VW4LL4QUGINGOXA, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.706935202 (2h17m14.487868603s) [noise=true , address=77.90.31.16:4746 , piece_id=AA75PBHV2E73ARBPZNUTZV7B5GGBXZUOVRL3VCEZ2EZ7HMUM234A, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.707554227 (2h17m14.488487618s) [noise=true , address=79.127.205.235:4685 , piece_id=RVZNRHTSFABHZ54X33PQVLFLMFMRM7VLA7SCXDVI5BZNRBEJUKIA, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.714121191 (2h17m14.495054578s) [noise=true , address=79.127.219.38:4001 , piece_id=O2JOUHJFKUGGFOFW6JRTQYIER5Z4DAPEOUGJCS3BTYDLERQFBEIA, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.714751905 (2h17m14.49568529s) [noise=true , address=104.36.82.47:4547 , piece_id=CC42C6HD572T657IQSBJDINSSXDUOWMMF4KEUTH64IEPGM5WFAIA, seg_pos={0 0}, upload=2101]: Uploading piece... Jul 27 00:08:58.811431908 (2h17m14.592365278s) [upload=2101]: Next returned nil segment Jul 27 00:08:58.811453600 (2h17m14.592386965s) [upload=2101]: Waiting for error group managing segments... Jul 27 00:08:58.869964784 (2h17m14.650898166s) [noise=true , address=77.90.31.16:4746 , piece_id=AA75PBHV2E73ARBPZNUTZV7B5GGBXZUOVRL3VCEZ2EZ7HMUM234A, seg_pos={0 0}, upload=2101]: Done uploading piece. err: <nil> Jul 27 00:08:58.885637487 (2h17m14.666570865s) [noise=true , address=79.127.219.38:4001 , piece_id=O2JOUHJFKUGGFOFW6JRTQYIER5Z4DAPEOUGJCS3BTYDLERQFBEIA, seg_pos={0 0}, upload=2101]: Done uploading piece. err: <nil> Jul 27 00:08:58.896591250 (2h17m14.677524628s) [noise=true , address=79.127.219.48:4074 , piece_id=RCKEXJFHLJTFCEK5CTIFWIPTT4OG7CCAHWJ5FZWVP2BMV6MF2ADA, seg_pos={0 0}, upload=2101]: Done uploading piece. err: <nil> Jul 27 00:08:59.285579378 (2h17m15.066512749s) [noise=true , address=79.127.205.245:4227 , piece_id=R33XV7QC723PQWKM5PR6QTVYBDBGUHVVMXQKM5ROSZ77W5J4JSAA, seg_pos={0 0}, upload=2101]: Done uploading piece. err: <nil> Jul 27 00:08:59.303912970 (2h17m15.084846332s) [noise=true , address=79.127.205.235:4685 , piece_id=RVZNRHTSFABHZ54X33PQVLFLMFMRM7VLA7SCXDVI5BZNRBEJUKIA, seg_pos={0 0}, upload=2101]: Done uploading piece. err: <nil> Jul 27 00:08:59.331426127 (2h17m15.112359494s) [noise=true , address=79.127.205.246:4281 , piece_id=WJ7SXHUVGEE36O7IKYTEUID3MBQ4BT6WEGSR5GOZ24WCOBF3FSPA, seg_pos={0 0}, upload=2101]: Done uploading piece. err: <nil> Jul 27 00:09:06.348746969 (2h17m22.129680336s) [noise=true , address=77.90.31.19:4856 , piece_id=VQQMI4YAFAPADAXDQWUUWVGMRSYEYBV2LCKXFBBOAEKBKSZ6TNWQ, seg_pos={0 0}, upload=2101]: Done uploading piece. err: <nil> Jul 27 00:09:07.955926224 (2h17m23.736859602s) [noise=true , address=64.49.13.138:4483 , piece_id=KKN7VFWOFSS6UKGMVP4O5SP34MM2KZDOCRQH7QFGD4YFQIE6YKDA, seg_pos={0 0}, upload=2101]: Done uploading piece. err: <nil> Jul 27 00:09:09.229263909 (2h17m25.010197304s) [noise=true , address=64.49.13.138:4482 , piece_id=A64AWGLBYK4RANPOSQDPEFFL5TNJ6HWRPIEPDYY4IT3DZM5J53ZQ, seg_pos={0 0}, upload=2101]: Done uploading piece. err: <nil> Jul 27 00:09:10.481553148 (2h17m26.262486523s) [noise=true , address=45.91.116.14:4206 , piece_id=H5BFYVDSTQKN2ZPRNOPFPS4EEUO6UY5H425OE4346QCHBBSL53CQ, seg_pos={0 0}, upload=2101]: Done uploading piece. err: <nil> Jul 27 00:09:58.589645966 (2h18m14.370579343s) [noise=true , address=104.36.82.55:4202 , piece_id=DPQYMMBYQZ4DM5IQLMZCIKF4HCGPAWRIVMDO3VPQ6SR4ZX37JIUQ, seg_pos={0 0}, upload=2101]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:104.36.82.55:4202): protocol: rpc: dial tcp 104.36.82.55:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:58.589750540 (2h18m14.370683918s) [noise=true , address=77.90.31.15:4714 , piece_id=3HZWM765U3XAABGE3NTW5JYHDVQNRXOIKUJU2R67FFYFZIWPMYKA, seg_pos={0 0}, upload=2101]: Done uploading piece. err: ecclient: upload failed (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP, address:77.90.31.15:4714): protocol: rpc: dial tcp 77.90.31.15:4714: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:58.589821519 (2h18m14.370754894s) [noise=true , address=104.36.82.56:4234 , piece_id=JX7MT6BQCEATXGPPHXGGGZTWPMS4MXURILRAH4TUBA6QJIPGCHNQ, seg_pos={0 0}, upload=2101]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:104.36.82.56:4234): protocol: write tcp 159.223.143.39:44494->104.36.82.56:4234: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:58.589879415 (2h18m14.370812779s) [noise=true , address=104.36.82.47:4547 , piece_id=CC42C6HD572T657IQSBJDINSSXDUOWMMF4KEUTH64IEPGM5WFAIA, seg_pos={0 0}, upload=2101]: Done uploading piece. err: ecclient: upload failed (node:12brQg9WQMKJgp1buz1wMS8W11XEwAx7RHG5h51EmLvwY5D5Hky, address:104.36.82.47:4547): protocol: write tcp 159.223.143.39:45666->104.36.82.47:4547: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:58.589955032 (2h18m14.370888408s) [noise=true , address=104.36.82.52:4107 , piece_id=C7RYDOFWHJSWRXUKO56UXWXTLZ5KNXKQFTAHIOQ5OIP34ESWS7PQ, seg_pos={0 0}, upload=2101]: Done uploading piece. err: ecclient: upload failed (node:12YVUtzGaKXr1Leb1URHZuaQXZv4zTtYwnv59j5GaAfd9zoK2Kj, address:104.36.82.52:4107): protocol: rpc: dial tcp 104.36.82.52:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:58.589997288 (2h18m14.370930671s) [noise=true , address=104.36.82.46:4513 , piece_id=OIAQWPQOYAYG3Q2BFKVCRYZPY5ACJ5M3RXMU6VW4LL4QUGINGOXA, seg_pos={0 0}, upload=2101]: Done uploading piece. err: ecclient: upload failed (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h, address:104.36.82.46:4513): protocol: write tcp 159.223.143.39:40684->104.36.82.46:4513: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:58.590036227 (2h18m14.370969605s) [noise=true , address=104.36.82.54:4169 , piece_id=PKI7VE3YP6M6NMCW4IA5UU7RMTEL4WLATOGAHAJAOYOPTWTVJJJQ, seg_pos={0 0}, upload=2101]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:58.590097110 (2h18m14.371030486s) [noise=true , address=77.90.31.15:4730 , piece_id=WVSI5HFXAUSKHLRAN4CASOGODTD4X73RDCOS4GRLRTNVUREUBY7A, seg_pos={0 0}, upload=2101]: Done uploading piece. err: ecclient: upload failed (node:1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, address:77.90.31.15:4730): protocol: write tcp 159.223.143.39:58880->77.90.31.15:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:58.590128313 (2h18m14.371061676s) [seg_pos={0 0}, upload=2101]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:09:58.615082988 (2h18m14.396016365s) [upload=2101]: Done uploading segments... Jul 27 00:09:58.615115575 (2h18m14.396048958s) [upload=2101]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.02727003s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 51 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX..] duration: 2.533230994s FAIL (2025-07-27 00:09:39 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.533264849s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [...X...X....XX..............X.X........X...XX...............] duration: 43.473840759s FAIL (2025-07-27 00:09:39 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 43.473863951s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX.] duration: 562.794377ms FAIL (2025-07-27 00:09:24 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== _id=OTZ6FQJRXFS3GWQIC2OZKBS4GR56LPUISPEDHWWQP7EONXTK4RZQ, seg_pos={0 0}, upload=2083]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:08:19.740998689 (2h16m35.521932116s) [noise=true , address=77.90.31.15:4699 , piece_id=LFCQEJG76VOFKXE2ZCGRXL7W2VNJL4TU4JYZU57F4DXJ55M7AYTQ, seg_pos={0 0}, upload=2083]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:08:19.741048351 (2h16m35.52198174s) [seg_pos={0 0}, upload=2083]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:08:19.764313990 (2h16m35.545247374s) [upload=2083]: Done uploading segments... Jul 27 00:08:19.764346822 (2h16m35.545280207s) [upload=2083]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:08:45.635298021 (2h17m1.4162315s) [upload=2097]: Starting upload Jul 27 00:08:45.635390125 (2h17m1.416323529s) [upload=2097]: Done starting upload Jul 27 00:08:45.635618718 (2h17m1.416552096s) [upload=2097]: Uploading segments... Jul 27 00:08:45.635780082 (2h17m1.416713462s) [upload=2097]: Got next segment. Inline: false Jul 27 00:08:45.635855437 (2h17m1.416788814s) [upload=2097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:08:45.635863070 (2h17m1.416796461s) [upload=2097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:08:45.666562989 (2h17m1.447496374s) [seg_pos={0 0}, upload=2097]: Begin upload segment... Jul 27 00:08:45.666633979 (2h17m1.447567347s) [seg_pos={0 0}, upload=2097]: Done begin upload segment. Jul 27 00:08:45.666674450 (2h17m1.447607829s) [noise=true , address=104.36.82.11:4115 , piece_id=FUAGHJHK7O64SCY2DIQ4ZH242XIELALGZFKHZBNDHADD6MNTSGXA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.666685892 (2h17m1.447619261s) [noise=true , address=45.91.116.5:4299 , piece_id=FLF5N2NDPNYOCYCNIGNWVY4WB2LURWFKXNAA7GNY6FOBSW3J64HA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.666754668 (2h17m1.447688044s) [noise=true , address=45.91.116.6:4975 , piece_id=DXOFPBUKYFW6GKR3V2UIYDIYLEIU3WCXL2ZIKUDNII6ABHSVDO6A, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667156517 (2h17m1.448089903s) [noise=true , address=45.91.116.4:4937 , piece_id=2KHWB574OKRBSHJYEM7VACSJNJDCDFXL6KLXBCXX7YU5M2PBPTAQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667216106 (2h17m1.448149484s) [noise=true , address=77.90.31.10:4518 , piece_id=7HCOYWWIANMJTHEJFMF6R67UALQWEHKDVGBD2XTHXNR4R2CGYSGQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667535834 (2h17m1.448469218s) [noise=true , address=45.91.116.8:4045 , piece_id=ZDGLNB566Q33PRSBA5CAYNCN76QWPJHMQMA2KGYIR3RRZR724V3Q, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667719777 (2h17m1.448653163s) [noise=true , address=45.91.116.10:4081 , piece_id=JX2VJ33Y7BXQSQAQ64RTKRQBCTI7VO6E4LQQGCSBZ7K5QVW2CO3Q, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667900825 (2h17m1.448834217s) [noise=true , address=77.90.31.14:4662 , piece_id=H6K6AETN3ZXM2SX7OYCTZEV5D5OXBNVEO2RZRGINGKSU7EFKCMVA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.668432972 (2h17m1.449366357s) [noise=true , address=104.36.82.13:4185 , piece_id=VUMNOYB6N5HVW7C3EG3IRYNF4K5GWYRAAFINFHAOLLDBUCAVO65Q, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.668922889 (2h17m1.449856273s) [noise=true , address=45.91.116.12:4225 , piece_id=UEJV3Q5Z74OJMIXHHDDXU7JX7SDZ2AMTVQAMBOT5ERJ4TJAXAXTA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.671253980 (2h17m1.452187358s) [noise=true , address=45.91.116.11:4117 , piece_id=X2AJPZ5QO4JM3E2S4NZ2RNZXUH5XC22NJ7YG63GJUV57T3GICPXA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.671603059 (2h17m1.452536438s) [noise=true , address=77.90.31.13:4628 , piece_id=H65YSRTR7DUDGFN7FMD3L64ENKLHADKZKDTU3XN6D64IHQUVEATQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.671890649 (2h17m1.452824036s) [noise=true , address=77.90.31.17:4772 , piece_id=6J7FD26JEJZUQLUXFIZIPFQ5WEJTLRMVEKCYLE744W32HK6RKDVQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.673442036 (2h17m1.454375416s) [noise=true , address=77.90.31.15:4698 , piece_id=GHPGFOMGZFLOAGWDOTZQWXISMJKUBO63TQ2CIOV653OUQYEUX5IQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.673811853 (2h17m1.454745223s) [noise=true , address=77.90.31.21:4879 , piece_id=4HXLD5QEEHXEQGYACGH6CNMH5VJGJP3434ICNU2WUGC44WQHG54A, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.686682455 (2h17m1.467615818s) [noise=true , address=104.36.82.5:4839 , piece_id=O3OAVQXCCBSNJBNGXSU2SZO5FZVRHUVBCY74KE2344PJY5OUGRRQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.771440822 (2h17m1.552374209s) [upload=2097]: Next returned nil segment Jul 27 00:08:45.771463691 (2h17m1.55239708s) [upload=2097]: Waiting for error group managing segments... Jul 27 00:08:45.775342955 (2h17m1.556276318s) [noise=true , address=45.91.116.4:4937 , piece_id=2KHWB574OKRBSHJYEM7VACSJNJDCDFXL6KLXBCXX7YU5M2PBPTAQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.776475944 (2h17m1.557409313s) [noise=true , address=45.91.116.5:4299 , piece_id=FLF5N2NDPNYOCYCNIGNWVY4WB2LURWFKXNAA7GNY6FOBSW3J64HA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.777015261 (2h17m1.557948628s) [noise=true , address=45.91.116.11:4117 , piece_id=X2AJPZ5QO4JM3E2S4NZ2RNZXUH5XC22NJ7YG63GJUV57T3GICPXA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.777591233 (2h17m1.55852462s) [noise=true , address=45.91.116.10:4081 , piece_id=JX2VJ33Y7BXQSQAQ64RTKRQBCTI7VO6E4LQQGCSBZ7K5QVW2CO3Q, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.778018122 (2h17m1.558951495s) [noise=true , address=45.91.116.8:4045 , piece_id=ZDGLNB566Q33PRSBA5CAYNCN76QWPJHMQMA2KGYIR3RRZR724V3Q, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.778418119 (2h17m1.559351488s) [noise=true , address=77.90.31.14:4662 , piece_id=H6K6AETN3ZXM2SX7OYCTZEV5D5OXBNVEO2RZRGINGKSU7EFKCMVA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.778929404 (2h17m1.559862779s) [noise=true , address=45.91.116.6:4975 , piece_id=DXOFPBUKYFW6GKR3V2UIYDIYLEIU3WCXL2ZIKUDNII6ABHSVDO6A, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.779537556 (2h17m1.560470931s) [noise=true , address=77.90.31.17:4772 , piece_id=6J7FD26JEJZUQLUXFIZIPFQ5WEJTLRMVEKCYLE744W32HK6RKDVQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.779812214 (2h17m1.560745582s) [noise=true , address=45.91.116.12:4225 , piece_id=UEJV3Q5Z74OJMIXHHDDXU7JX7SDZ2AMTVQAMBOT5ERJ4TJAXAXTA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:46.684702430 (2h17m2.465635798s) [noise=true , address=77.90.31.21:4879 , piece_id=4HXLD5QEEHXEQGYACGH6CNMH5VJGJP3434ICNU2WUGC44WQHG54A, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:47.073910770 (2h17m2.854844136s) [noise=true , address=77.90.31.10:4518 , piece_id=7HCOYWWIANMJTHEJFMF6R67UALQWEHKDVGBD2XTHXNR4R2CGYSGQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:47.166542112 (2h17m2.947475475s) [noise=true , address=77.90.31.13:4628 , piece_id=H65YSRTR7DUDGFN7FMD3L64ENKLHADKZKDTU3XN6D64IHQUVEATQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:09:15.606542690 (2h17m31.387476056s) [noise=true , address=104.36.82.5:4839 , piece_id=O3OAVQXCCBSNJBNGXSU2SZO5FZVRHUVBCY74KE2344PJY5OUGRRQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:15.606697150 (2h17m31.387630526s) [noise=true , address=77.90.31.15:4698 , piece_id=GHPGFOMGZFLOAGWDOTZQWXISMJKUBO63TQ2CIOV653OUQYEUX5IQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:15.606735183 (2h17m31.387668552s) [noise=true , address=104.36.82.13:4185 , piece_id=VUMNOYB6N5HVW7C3EG3IRYNF4K5GWYRAAFINFHAOLLDBUCAVO65Q, seg_pos={0 0}, upload=2097]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 159.223.143.39:37514->104.36.82.13:4185: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:15.606756595 (2h17m31.387689958s) [noise=true , address=104.36.82.11:4115 , piece_id=FUAGHJHK7O64SCY2DIQ4ZH242XIELALGZFKHZBNDHADD6MNTSGXA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:15.606778960 (2h17m31.387712338s) [seg_pos={0 0}, upload=2097]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:09:15.635367371 (2h17m31.416300738s) [upload=2097]: Done uploading segments... Jul 27 00:09:15.635393090 (2h17m31.416326454s) [upload=2097]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 562.893069ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..XXX...XX.....X.XXX...............X..X......X.XX....X...X..] duration: 483.022918ms FAIL (2025-07-27 00:09:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 483.044551ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX.] duration: 1.867645699s FAIL (2025-07-27 00:09:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 1.86769198s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 54 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX] duration: 30.031054088s FAIL (2025-07-27 00:09:15 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== _id=OTZ6FQJRXFS3GWQIC2OZKBS4GR56LPUISPEDHWWQP7EONXTK4RZQ, seg_pos={0 0}, upload=2083]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:08:19.740998689 (2h16m35.521932116s) [noise=true , address=77.90.31.15:4699 , piece_id=LFCQEJG76VOFKXE2ZCGRXL7W2VNJL4TU4JYZU57F4DXJ55M7AYTQ, seg_pos={0 0}, upload=2083]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:08:19.741048351 (2h16m35.52198174s) [seg_pos={0 0}, upload=2083]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:08:19.764313990 (2h16m35.545247374s) [upload=2083]: Done uploading segments... Jul 27 00:08:19.764346822 (2h16m35.545280207s) [upload=2083]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:08:45.635298021 (2h17m1.4162315s) [upload=2097]: Starting upload Jul 27 00:08:45.635390125 (2h17m1.416323529s) [upload=2097]: Done starting upload Jul 27 00:08:45.635618718 (2h17m1.416552096s) [upload=2097]: Uploading segments... Jul 27 00:08:45.635780082 (2h17m1.416713462s) [upload=2097]: Got next segment. Inline: false Jul 27 00:08:45.635855437 (2h17m1.416788814s) [upload=2097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:08:45.635863070 (2h17m1.416796461s) [upload=2097]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:08:45.666562989 (2h17m1.447496374s) [seg_pos={0 0}, upload=2097]: Begin upload segment... Jul 27 00:08:45.666633979 (2h17m1.447567347s) [seg_pos={0 0}, upload=2097]: Done begin upload segment. Jul 27 00:08:45.666674450 (2h17m1.447607829s) [noise=true , address=104.36.82.11:4115 , piece_id=FUAGHJHK7O64SCY2DIQ4ZH242XIELALGZFKHZBNDHADD6MNTSGXA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.666685892 (2h17m1.447619261s) [noise=true , address=45.91.116.5:4299 , piece_id=FLF5N2NDPNYOCYCNIGNWVY4WB2LURWFKXNAA7GNY6FOBSW3J64HA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.666754668 (2h17m1.447688044s) [noise=true , address=45.91.116.6:4975 , piece_id=DXOFPBUKYFW6GKR3V2UIYDIYLEIU3WCXL2ZIKUDNII6ABHSVDO6A, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667156517 (2h17m1.448089903s) [noise=true , address=45.91.116.4:4937 , piece_id=2KHWB574OKRBSHJYEM7VACSJNJDCDFXL6KLXBCXX7YU5M2PBPTAQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667216106 (2h17m1.448149484s) [noise=true , address=77.90.31.10:4518 , piece_id=7HCOYWWIANMJTHEJFMF6R67UALQWEHKDVGBD2XTHXNR4R2CGYSGQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667535834 (2h17m1.448469218s) [noise=true , address=45.91.116.8:4045 , piece_id=ZDGLNB566Q33PRSBA5CAYNCN76QWPJHMQMA2KGYIR3RRZR724V3Q, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667719777 (2h17m1.448653163s) [noise=true , address=45.91.116.10:4081 , piece_id=JX2VJ33Y7BXQSQAQ64RTKRQBCTI7VO6E4LQQGCSBZ7K5QVW2CO3Q, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.667900825 (2h17m1.448834217s) [noise=true , address=77.90.31.14:4662 , piece_id=H6K6AETN3ZXM2SX7OYCTZEV5D5OXBNVEO2RZRGINGKSU7EFKCMVA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.668432972 (2h17m1.449366357s) [noise=true , address=104.36.82.13:4185 , piece_id=VUMNOYB6N5HVW7C3EG3IRYNF4K5GWYRAAFINFHAOLLDBUCAVO65Q, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.668922889 (2h17m1.449856273s) [noise=true , address=45.91.116.12:4225 , piece_id=UEJV3Q5Z74OJMIXHHDDXU7JX7SDZ2AMTVQAMBOT5ERJ4TJAXAXTA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.671253980 (2h17m1.452187358s) [noise=true , address=45.91.116.11:4117 , piece_id=X2AJPZ5QO4JM3E2S4NZ2RNZXUH5XC22NJ7YG63GJUV57T3GICPXA, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.671603059 (2h17m1.452536438s) [noise=true , address=77.90.31.13:4628 , piece_id=H65YSRTR7DUDGFN7FMD3L64ENKLHADKZKDTU3XN6D64IHQUVEATQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.671890649 (2h17m1.452824036s) [noise=true , address=77.90.31.17:4772 , piece_id=6J7FD26JEJZUQLUXFIZIPFQ5WEJTLRMVEKCYLE744W32HK6RKDVQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.673442036 (2h17m1.454375416s) [noise=true , address=77.90.31.15:4698 , piece_id=GHPGFOMGZFLOAGWDOTZQWXISMJKUBO63TQ2CIOV653OUQYEUX5IQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.673811853 (2h17m1.454745223s) [noise=true , address=77.90.31.21:4879 , piece_id=4HXLD5QEEHXEQGYACGH6CNMH5VJGJP3434ICNU2WUGC44WQHG54A, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.686682455 (2h17m1.467615818s) [noise=true , address=104.36.82.5:4839 , piece_id=O3OAVQXCCBSNJBNGXSU2SZO5FZVRHUVBCY74KE2344PJY5OUGRRQ, seg_pos={0 0}, upload=2097]: Uploading piece... Jul 27 00:08:45.771440822 (2h17m1.552374209s) [upload=2097]: Next returned nil segment Jul 27 00:08:45.771463691 (2h17m1.55239708s) [upload=2097]: Waiting for error group managing segments... Jul 27 00:08:45.775342955 (2h17m1.556276318s) [noise=true , address=45.91.116.4:4937 , piece_id=2KHWB574OKRBSHJYEM7VACSJNJDCDFXL6KLXBCXX7YU5M2PBPTAQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.776475944 (2h17m1.557409313s) [noise=true , address=45.91.116.5:4299 , piece_id=FLF5N2NDPNYOCYCNIGNWVY4WB2LURWFKXNAA7GNY6FOBSW3J64HA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.777015261 (2h17m1.557948628s) [noise=true , address=45.91.116.11:4117 , piece_id=X2AJPZ5QO4JM3E2S4NZ2RNZXUH5XC22NJ7YG63GJUV57T3GICPXA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.777591233 (2h17m1.55852462s) [noise=true , address=45.91.116.10:4081 , piece_id=JX2VJ33Y7BXQSQAQ64RTKRQBCTI7VO6E4LQQGCSBZ7K5QVW2CO3Q, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.778018122 (2h17m1.558951495s) [noise=true , address=45.91.116.8:4045 , piece_id=ZDGLNB566Q33PRSBA5CAYNCN76QWPJHMQMA2KGYIR3RRZR724V3Q, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.778418119 (2h17m1.559351488s) [noise=true , address=77.90.31.14:4662 , piece_id=H6K6AETN3ZXM2SX7OYCTZEV5D5OXBNVEO2RZRGINGKSU7EFKCMVA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.778929404 (2h17m1.559862779s) [noise=true , address=45.91.116.6:4975 , piece_id=DXOFPBUKYFW6GKR3V2UIYDIYLEIU3WCXL2ZIKUDNII6ABHSVDO6A, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.779537556 (2h17m1.560470931s) [noise=true , address=77.90.31.17:4772 , piece_id=6J7FD26JEJZUQLUXFIZIPFQ5WEJTLRMVEKCYLE744W32HK6RKDVQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:45.779812214 (2h17m1.560745582s) [noise=true , address=45.91.116.12:4225 , piece_id=UEJV3Q5Z74OJMIXHHDDXU7JX7SDZ2AMTVQAMBOT5ERJ4TJAXAXTA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:46.684702430 (2h17m2.465635798s) [noise=true , address=77.90.31.21:4879 , piece_id=4HXLD5QEEHXEQGYACGH6CNMH5VJGJP3434ICNU2WUGC44WQHG54A, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:47.073910770 (2h17m2.854844136s) [noise=true , address=77.90.31.10:4518 , piece_id=7HCOYWWIANMJTHEJFMF6R67UALQWEHKDVGBD2XTHXNR4R2CGYSGQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:08:47.166542112 (2h17m2.947475475s) [noise=true , address=77.90.31.13:4628 , piece_id=H65YSRTR7DUDGFN7FMD3L64ENKLHADKZKDTU3XN6D64IHQUVEATQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: <nil> Jul 27 00:09:15.606542690 (2h17m31.387476056s) [noise=true , address=104.36.82.5:4839 , piece_id=O3OAVQXCCBSNJBNGXSU2SZO5FZVRHUVBCY74KE2344PJY5OUGRRQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:15.606697150 (2h17m31.387630526s) [noise=true , address=77.90.31.15:4698 , piece_id=GHPGFOMGZFLOAGWDOTZQWXISMJKUBO63TQ2CIOV653OUQYEUX5IQ, seg_pos={0 0}, upload=2097]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:15.606735183 (2h17m31.387668552s) [noise=true , address=104.36.82.13:4185 , piece_id=VUMNOYB6N5HVW7C3EG3IRYNF4K5GWYRAAFINFHAOLLDBUCAVO65Q, seg_pos={0 0}, upload=2097]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 159.223.143.39:37514->104.36.82.13:4185: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:15.606756595 (2h17m31.387689958s) [noise=true , address=104.36.82.11:4115 , piece_id=FUAGHJHK7O64SCY2DIQ4ZH242XIELALGZFKHZBNDHADD6MNTSGXA, seg_pos={0 0}, upload=2097]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:15.606778960 (2h17m31.387712338s) [seg_pos={0 0}, upload=2097]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:09:15.635367371 (2h17m31.416300738s) [upload=2097]: Done uploading segments... Jul 27 00:09:15.635393090 (2h17m31.416326454s) [upload=2097]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.031091737s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.035207603s FAIL (2025-07-27 00:08:58 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== pb.BatchRequestItem_SegmentBegin Jul 27 00:07:58.571471335 (2h16m14.352404726s) [seg_pos={0 0}, upload=2086]: Begin upload segment... Jul 27 00:07:58.571528992 (2h16m14.352462372s) [seg_pos={0 0}, upload=2086]: Done begin upload segment. Jul 27 00:07:58.571560811 (2h16m14.352494193s) [noise=true , address=45.91.116.14:4206 , piece_id=PJKH4TLQXKTOCCH3GAZVCFNM54SARIP3DTL4W4R6OGWIDL6C6EBQ, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.571880479 (2h16m14.352813863s) [noise=true , address=79.127.205.245:4246 , piece_id=HFQKIFWQGBQIBBG5TFMSGY6SEEGVAU4BBDNVSBNINHBHLWMO4TPA, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.572151788 (2h16m14.353085178s) [noise=true , address=79.127.205.234:4605 , piece_id=SZENAG67GU2AXNZ3CIOOQQD3UC7RV2SKMVJ24BHZJNUQVN6T5ZMQ, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.572719939 (2h16m14.353653324s) [noise=true , address=79.127.220.98:4422 , piece_id=UIKENZ43BPVLCJJ6WRQGVPZORCKGW36RKMIN23FPGSGNFKP55UJA, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.572803014 (2h16m14.353736403s) [noise=true , address=79.127.205.246:4283 , piece_id=KE5SLI2DBHGYIP72NCHS7674KCFLRP5K775AJHVVEHLGARLRVIXA, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.573291054 (2h16m14.354224464s) [noise=true , address=79.127.220.97:4361 , piece_id=RMYNTMR6W34NEYXLUWDNQLS4LXSHZOGMPPJTOOU3QMHEGXA2BFNA, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.573300466 (2h16m14.354233848s) [noise=true , address=77.90.31.16:4743 , piece_id=OLKUPFXZCEV7FKHOVUSMNVM2RKOP2PYMZ2PUDXHLLSXKGCH2ZW2Q, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.573752384 (2h16m14.354685768s) [noise=true , address=104.36.82.53:4144 , piece_id=ZRLN4TXEOQVSBICQZFS3DPOSYOISCKSW6GCQFM42FULQM3XZG63Q, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.573885407 (2h16m14.354818793s) [noise=true , address=79.127.201.215:4351 , piece_id=5JVRJOR6A2TH7WMEE647K3APUK4XJODB23MU2HSI6KHVTK6XCZYA, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.574210503 (2h16m14.355143895s) [noise=true , address=77.90.31.19:4871 , piece_id=WRAKP2KI5M5L4URDPI3KUHPCTIIHKRFNZ6VUN32H7ELEWUEMTRVQ, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.574428443 (2h16m14.355361825s) [noise=true , address=104.36.82.56:4257 , piece_id=OJFFPWZ2CGHSD76Z73XUIYVUITPGKYP6P7EFONOYXOGUDX45ZUIA, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.574640678 (2h16m14.355574057s) [noise=true , address=45.91.116.9:4277 , piece_id=THCPO5GM4BVJPQILOVPYHOVUHESWFB4AU34Q435NRCYKNRQP54KQ, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.575136064 (2h16m14.356069446s) [noise=true , address=77.90.31.13:4653 , piece_id=MEFACJFVOZR27ENWMH4G7WDV2FRY36KDY6T2CCPX2XPWQJLC2U5A, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.575235618 (2h16m14.356169003s) [noise=true , address=104.36.82.46:4501 , piece_id=CHWEXONG3ZVPDQWHLCJZXIHPY6TKEWDUAKGPYR2EOXRYV4GF3DGQ, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.575573434 (2h16m14.356506913s) [noise=true , address=77.90.31.18:4823 , piece_id=Y5FMCBEYBTDR6QHWIIZS7YP26B2L45IXKONF6FOL2CY2CPILD5WQ, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.575753558 (2h16m14.356686938s) [noise=true , address=104.36.82.47:4527 , piece_id=7KD6WHTAZVDHKZLDYNGHZF7KQABMOJKY3EY3LKXGSUIJUSSFOVWQ, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.576239310 (2h16m14.35717269s) [noise=true , address=104.36.82.52:4114 , piece_id=VMMLOIGHHIKR3YTCHZ7Z7TGZ53J52FYY3LVJT3X3HJL4PVB7JAMQ, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.576771158 (2h16m14.357704539s) [noise=true , address=77.90.31.21:4909 , piece_id=DZJLMYK72PGPOWIXATBXEYBTBAGPWQV3KPTKXWVKUWKDJNEDPY6A, seg_pos={0 0}, upload=2086]: Uploading piece... Jul 27 00:07:58.656387892 (2h16m14.437321259s) [upload=2086]: Next returned nil segment Jul 27 00:07:58.656405692 (2h16m14.437339058s) [upload=2086]: Waiting for error group managing segments... Jul 27 00:07:58.661053248 (2h16m14.441986624s) [noise=true , address=77.90.31.16:4743 , piece_id=OLKUPFXZCEV7FKHOVUSMNVM2RKOP2PYMZ2PUDXHLLSXKGCH2ZW2Q, seg_pos={0 0}, upload=2086]: Done uploading piece. err: <nil> Jul 27 00:07:58.689500879 (2h16m14.470434254s) [noise=true , address=79.127.220.97:4361 , piece_id=RMYNTMR6W34NEYXLUWDNQLS4LXSHZOGMPPJTOOU3QMHEGXA2BFNA, seg_pos={0 0}, upload=2086]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 27 00:07:58.824486404 (2h16m14.605419784s) [noise=true , address=77.90.31.18:4823 , piece_id=Y5FMCBEYBTDR6QHWIIZS7YP26B2L45IXKONF6FOL2CY2CPILD5WQ, seg_pos={0 0}, upload=2086]: Done uploading piece. err: <nil> Jul 27 00:07:58.826808475 (2h16m14.607741843s) [noise=true , address=77.90.31.21:4909 , piece_id=DZJLMYK72PGPOWIXATBXEYBTBAGPWQV3KPTKXWVKUWKDJNEDPY6A, seg_pos={0 0}, upload=2086]: Done uploading piece. err: <nil> Jul 27 00:07:58.911693819 (2h16m14.692627205s) [noise=true , address=45.91.116.9:4277 , piece_id=THCPO5GM4BVJPQILOVPYHOVUHESWFB4AU34Q435NRCYKNRQP54KQ, seg_pos={0 0}, upload=2086]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 27 00:07:59.160351458 (2h16m14.941284823s) [noise=true , address=79.127.205.245:4246 , piece_id=HFQKIFWQGBQIBBG5TFMSGY6SEEGVAU4BBDNVSBNINHBHLWMO4TPA, seg_pos={0 0}, upload=2086]: Done uploading piece. err: <nil> Jul 27 00:07:59.163461136 (2h16m14.944394513s) [noise=true , address=79.127.205.234:4605 , piece_id=SZENAG67GU2AXNZ3CIOOQQD3UC7RV2SKMVJ24BHZJNUQVN6T5ZMQ, seg_pos={0 0}, upload=2086]: Done uploading piece. err: <nil> Jul 27 00:07:59.179814556 (2h16m14.960747926s) [noise=true , address=79.127.205.246:4283 , piece_id=KE5SLI2DBHGYIP72NCHS7674KCFLRP5K775AJHVVEHLGARLRVIXA, seg_pos={0 0}, upload=2086]: Done uploading piece. err: <nil> Jul 27 00:08:00.141739984 (2h16m15.922673364s) [noise=true , address=79.127.220.98:4422 , piece_id=UIKENZ43BPVLCJJ6WRQGVPZORCKGW36RKMIN23FPGSGNFKP55UJA, seg_pos={0 0}, upload=2086]: Done uploading piece. err: <nil> Jul 27 00:08:02.644197060 (2h16m18.425130438s) [noise=true , address=77.90.31.19:4871 , piece_id=WRAKP2KI5M5L4URDPI3KUHPCTIIHKRFNZ6VUN32H7ELEWUEMTRVQ, seg_pos={0 0}, upload=2086]: Done uploading piece. err: <nil> Jul 27 00:08:06.978277581 (2h16m22.75921095s) [noise=true , address=45.91.116.14:4206 , piece_id=PJKH4TLQXKTOCCH3GAZVCFNM54SARIP3DTL4W4R6OGWIDL6C6EBQ, seg_pos={0 0}, upload=2086]: Done uploading piece. err: <nil> Jul 27 00:08:31.041151596 (2h16m46.822084965s) [noise=true , address=79.127.201.215:4351 , piece_id=5JVRJOR6A2TH7WMEE647K3APUK4XJODB23MU2HSI6KHVTK6XCZYA, seg_pos={0 0}, upload=2086]: Done uploading piece. err: <nil> Jul 27 00:08:54.070821324 (2h17m9.851754702s) [noise=true , address=77.90.31.13:4653 , piece_id=MEFACJFVOZR27ENWMH4G7WDV2FRY36KDY6T2CCPX2XPWQJLC2U5A, seg_pos={0 0}, upload=2086]: Done uploading piece. err: <nil> Jul 27 00:08:58.485000935 (2h17m14.265934317s) [noise=true , address=104.36.82.47:4527 , piece_id=7KD6WHTAZVDHKZLDYNGHZF7KQABMOJKY3EY3LKXGSUIJUSSFOVWQ, seg_pos={0 0}, upload=2086]: Done uploading piece. err: ecclient: upload failed (node:17xmkbFoj3PTJnAygsjRAcfdd17AfSkPP2xdZp68GGPrvfMuU6, address:104.36.82.47:4527): protocol: write tcp 159.223.143.39:45842->104.36.82.47:4527: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:58.485126710 (2h17m14.266060089s) [noise=true , address=104.36.82.46:4501 , piece_id=CHWEXONG3ZVPDQWHLCJZXIHPY6TKEWDUAKGPYR2EOXRYV4GF3DGQ, seg_pos={0 0}, upload=2086]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:104.36.82.46:4501): protocol: write tcp 159.223.143.39:47962->104.36.82.46:4501: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:58.486013090 (2h17m14.266946472s) [noise=true , address=104.36.82.56:4257 , piece_id=OJFFPWZ2CGHSD76Z73XUIYVUITPGKYP6P7EFONOYXOGUDX45ZUIA, seg_pos={0 0}, upload=2086]: Done uploading piece. err: ecclient: upload failed (node:12iw1KEPBAYhvSachvkigv7bPXfGN5XEugvxGC5YTBV2Muymir3, address:104.36.82.56:4257): protocol: write tcp 159.223.143.39:51022->104.36.82.56:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:58.486104889 (2h17m14.267038257s) [noise=true , address=104.36.82.52:4114 , piece_id=VMMLOIGHHIKR3YTCHZ7Z7TGZ53J52FYY3LVJT3X3HJL4PVB7JAMQ, seg_pos={0 0}, upload=2086]: Done uploading piece. err: ecclient: upload failed (node:12PXReFMGqAhCAp5BSE3vsJdfRAUathsJLdB8Guz7DzW6UHmATM, address:104.36.82.52:4114): protocol: write tcp 159.223.143.39:34840->104.36.82.52:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:58.486174191 (2h17m14.267107557s) [noise=true , address=104.36.82.53:4144 , piece_id=ZRLN4TXEOQVSBICQZFS3DPOSYOISCKSW6GCQFM42FULQM3XZG63Q, seg_pos={0 0}, upload=2086]: Done uploading piece. err: ecclient: upload failed (node:1FfzcPNLmNdrvoZBKcmCLz8d9rYSAnjFTo4NBnxBzN6dVcTNwt, address:104.36.82.53:4144): protocol: write tcp 159.223.143.39:37018->104.36.82.53:4144: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:58.486197912 (2h17m14.267131297s) [seg_pos={0 0}, upload=2086]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 00:08:58.515202173 (2h17m14.296135543s) [upload=2086]: Done uploading segments... Jul 27 00:08:58.515241979 (2h17m14.296175358s) [upload=2086]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration:* 1m0.035233581s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX] duration: 30.027115385s FAIL (2025-07-27 00:08:52 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.027173922s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [....X...X....XX..............X.X........X...XX..............] duration: 29.871730396s FAIL (2025-07-27 00:08:34 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.871763063s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 54 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....X] duration: 30.027033025s FAIL (2025-07-27 00:08:19 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 528133661s) [noise=true , address=104.36.82.5:4837 , piece_id=K3P6BF7XD4U6ASNOX6O625GZQUCAYF6GKTNNAPJLICBHRYGZJDRA, seg_pos={0 0}, upload=2063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: rpc: dial tcp 104.36.82.5:4837: operation was canceled; conn closed: op; piecestore: piecestore close: context canceled Jul 27 00:06:38.747231402 (2h14m54.528164781s) [seg_pos={0 0}, upload=2063]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:06:38.747323302 (2h14m54.528256679s) [upload=2063]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:06:38.747340233 (2h14m54.528273609s) [upload=2063]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:06:38.884500181 (2h14m54.665433546s) [upload=2063]: Done uploading segments... Jul 27 00:06:38.884518081 (2h14m54.66545145s) [upload=2063]: Upload finished. err: <nil> Jul 27 00:07:49.767137327 (2h16m5.548070693s) [upload=2083]: Starting upload Jul 27 00:07:49.767257443 (2h16m5.548190826s) [upload=2083]: Done starting upload Jul 27 00:07:49.767387204 (2h16m5.548320576s) [upload=2083]: Uploading segments... Jul 27 00:07:49.767698085 (2h16m5.548631462s) [upload=2083]: Got next segment. Inline: false Jul 27 00:07:49.767782455 (2h16m5.548715837s) [upload=2083]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:07:49.767788986 (2h16m5.548722368s) [upload=2083]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:07:49.798756949 (2h16m5.579690327s) [seg_pos={0 0}, upload=2083]: Begin upload segment... Jul 27 00:07:49.798814689 (2h16m5.579748074s) [seg_pos={0 0}, upload=2083]: Done begin upload segment. Jul 27 00:07:49.798842192 (2h16m5.57977556s) [noise=true , address=77.90.31.18:4806 , piece_id=H4UA5JHVM23TDJUZ5DR4WW26EIZVTDAT3RVABVWEMZYIUUR2OQPQ, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.799356728 (2h16m5.580290114s) [noise=true , address=104.36.82.16:4294 , piece_id=YWMX6IILJBNO3E2R2OHCWX6YOKJUDECASJU3LEOMQWCPDBCKRZRQ, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.799709141 (2h16m5.580642522s) [noise=true , address=45.91.116.5:4297 , piece_id=6V5KPO2RD74DPAVUEBJVPKLCVGMXUDMOWHHQR63MEMD5INUFZUTQ, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.800006755 (2h16m5.580940131s) [noise=true , address=104.36.82.10:4077 , piece_id=OTZ6FQJRXFS3GWQIC2OZKBS4GR56LPUISPEDHWWQP7EONXTK4RZQ, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.800439748 (2h16m5.58137313s) [noise=true , address=104.36.82.13:4186 , piece_id=3RGHTNMVOVQEMPZK4BBH3ZVAVMPJTINCMCFCM2VRSQNQDKLG6YAA, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.800720006 (2h16m5.581653384s) [noise=true , address=77.90.31.18:4807 , piece_id=MJJ22ZI6ZTFOTZK67DHJ2OQSQN63QU4IKLT4DYI3PQULB36HZYPA, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.801015426 (2h16m5.581948804s) [noise=true , address=77.90.31.12:4589 , piece_id=HZQC3IPR2XYKIGQXJNB3TSZPZAZAHCI7UGFPTUIXFIREGB34WPWQ, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.801289990 (2h16m5.582223366s) [noise=true , address=77.90.31.14:4661 , piece_id=EZEFYOBG3JCFOEICV5MFCGBVLJBJDIABGJDMBNHMMPUJDEYKHM3Q, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.801649561 (2h16m5.582582939s) [noise=true , address=77.90.31.19:4841 , piece_id=YBYPNCG5SSNEPAIAMRJANLLCERMEAVI33GBOZO566U3DZZUI3D4A, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.801972652 (2h16m5.58290603s) [noise=true , address=77.90.31.15:4699 , piece_id=LFCQEJG76VOFKXE2ZCGRXL7W2VNJL4TU4JYZU57F4DXJ55M7AYTQ, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.802262026 (2h16m5.583195408s) [noise=true , address=104.36.82.6:4875 , piece_id=NMBAHTMLNMDCJYB3TBNT2SEAJRJBN37UMURMD7IV7IARPNPU2SZA, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.802658158 (2h16m5.583591537s) [noise=true , address=104.36.82.7:4912 , piece_id=FGKK7XKYCTCHZH47Y7EKEVBUGMLGVRBNBIWWQGMR6BPB5QIXKM2Q, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.803108676 (2h16m5.584042064s) [noise=true , address=45.91.116.12:4225 , piece_id=PE62HL7CFXWLHTIXWJ3PJJK5WSNKHWS7HDZUGVS3BV2FDCSOFTCA, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.803563260 (2h16m5.584496643s) [noise=true , address=45.91.116.8:4048 , piece_id=PNZ7UI7N4R3IAWH47QAFIOWISWQDDZCLU7K5Y7DHS6OU7OX36EHA, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.803878613 (2h16m5.584811998s) [noise=true , address=45.91.116.10:4083 , piece_id=ZG3MJCKUA6EKYJHDDA4AFB3D3IS6JKBKM775YUUTBIN6DLBVC4XA, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.804211708 (2h16m5.585145086s) [noise=true , address=45.91.116.6:4974 , piece_id=DDGF7EDB5SBJ4H6YIYHYRFPEQ4ELLU6N272ZOKPTMF6VFRCDPVZA, seg_pos={0 0}, upload=2083]: Uploading piece... Jul 27 00:07:49.888520700 (2h16m5.669454068s) [upload=2083]: Next returned nil segment Jul 27 00:07:49.888540905 (2h16m5.669474287s) [upload=2083]: Waiting for error group managing segments... Jul 27 00:07:49.892980242 (2h16m5.673913611s) [noise=true , address=45.91.116.6:4974 , piece_id=DDGF7EDB5SBJ4H6YIYHYRFPEQ4ELLU6N272ZOKPTMF6VFRCDPVZA, seg_pos={0 0}, upload=2083]: Done uploading piece. err: <nil> Jul 27 00:07:49.893437036 (2h16m5.674370404s) [noise=true , address=45.91.116.12:4225 , piece_id=PE62HL7CFXWLHTIXWJ3PJJK5WSNKHWS7HDZUGVS3BV2FDCSOFTCA, seg_pos={0 0}, upload=2083]: Done uploading piece. err: <nil> Jul 27 00:07:49.893674407 (2h16m5.674607776s) [noise=true , address=77.90.31.14:4661 , piece_id=EZEFYOBG3JCFOEICV5MFCGBVLJBJDIABGJDMBNHMMPUJDEYKHM3Q, seg_pos={0 0}, upload=2083]: Done uploading piece. err: <nil> Jul 27 00:07:49.893865011 (2h16m5.674798375s) [noise=true , address=45.91.116.8:4048 , piece_id=PNZ7UI7N4R3IAWH47QAFIOWISWQDDZCLU7K5Y7DHS6OU7OX36EHA, seg_pos={0 0}, upload=2083]: Done uploading piece. err: <nil> Jul 27 00:07:49.894269534 (2h16m5.675202896s) [noise=true , address=45.91.116.5:4297 , piece_id=6V5KPO2RD74DPAVUEBJVPKLCVGMXUDMOWHHQR63MEMD5INUFZUTQ, seg_pos={0 0}, upload=2083]: Done uploading piece. err: <nil> Jul 27 00:07:49.941852294 (2h16m5.722785663s) [noise=true , address=45.91.116.10:4083 , piece_id=ZG3MJCKUA6EKYJHDDA4AFB3D3IS6JKBKM775YUUTBIN6DLBVC4XA, seg_pos={0 0}, upload=2083]: Done uploading piece. err: <nil> Jul 27 00:07:51.292263230 (2h16m7.07319661s) [noise=true , address=77.90.31.19:4841 , piece_id=YBYPNCG5SSNEPAIAMRJANLLCERMEAVI33GBOZO566U3DZZUI3D4A, seg_pos={0 0}, upload=2083]: Done uploading piece. err: <nil> Jul 27 00:07:51.299811565 (2h16m7.080744926s) [noise=true , address=77.90.31.18:4806 , piece_id=H4UA5JHVM23TDJUZ5DR4WW26EIZVTDAT3RVABVWEMZYIUUR2OQPQ, seg_pos={0 0}, upload=2083]: Done uploading piece. err: <nil> Jul 27 00:07:51.311794906 (2h16m7.092728283s) [noise=true , address=77.90.31.18:4807 , piece_id=MJJ22ZI6ZTFOTZK67DHJ2OQSQN63QU4IKLT4DYI3PQULB36HZYPA, seg_pos={0 0}, upload=2083]: Done uploading piece. err: <nil> Jul 27 00:07:51.322780770 (2h16m7.103714378s) [noise=true , address=77.90.31.12:4589 , piece_id=HZQC3IPR2XYKIGQXJNB3TSZPZAZAHCI7UGFPTUIXFIREGB34WPWQ, seg_pos={0 0}, upload=2083]: Done uploading piece. err: <nil> Jul 27 00:08:19.739874014 (2h16m35.520807413s) [noise=true , address=104.36.82.6:4875 , piece_id=NMBAHTMLNMDCJYB3TBNT2SEAJRJBN37UMURMD7IV7IARPNPU2SZA, seg_pos={0 0}, upload=2083]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:08:19.740039031 (2h16m35.52097241s) [noise=true , address=104.36.82.13:4186 , piece_id=3RGHTNMVOVQEMPZK4BBH3ZVAVMPJTINCMCFCM2VRSQNQDKLG6YAA, seg_pos={0 0}, upload=2083]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:08:19.740276156 (2h16m35.521209536s) [noise=true , address=104.36.82.16:4294 , piece_id=YWMX6IILJBNO3E2R2OHCWX6YOKJUDECASJU3LEOMQWCPDBCKRZRQ, seg_pos={0 0}, upload=2083]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:08:19.740653885 (2h16m35.52158727s) [noise=true , address=104.36.82.7:4912 , piece_id=FGKK7XKYCTCHZH47Y7EKEVBUGMLGVRBNBIWWQGMR6BPB5QIXKM2Q, seg_pos={0 0}, upload=2083]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:08:19.740760999 (2h16m35.521694375s) [noise=true , address=104.36.82.10:4077 , piece_id=OTZ6FQJRXFS3GWQIC2OZKBS4GR56LPUISPEDHWWQP7EONXTK4RZQ, seg_pos={0 0}, upload=2083]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:08:19.740998689 (2h16m35.521932116s) [noise=true , address=77.90.31.15:4699 , piece_id=LFCQEJG76VOFKXE2ZCGRXL7W2VNJL4TU4JYZU57F4DXJ55M7AYTQ, seg_pos={0 0}, upload=2083]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:08:19.741048351 (2h16m35.52198174s) [seg_pos={0 0}, upload=2083]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:08:19.764313990 (2h16m35.545247374s) [upload=2083]: Done uploading segments... Jul 27 00:08:19.764346822 (2h16m35.545280207s) [upload=2083]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.027078164s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...XXX...XX.....X.XXX...............X..X......X.XX....X...X.] duration: 1.463660392s FAIL (2025-07-27 00:08:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.463683774s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)