Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................XX.................] duration: 665.59893ms FAIL (2025-07-26 20:40:26 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 665.674934ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 316.828691ms FAIL (2025-07-26 20:40:08 UTC) orbiter warning: uplink: too many requests Duration: 316.866487ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................XX..............] duration: 501.491075ms FAIL (2025-07-26 20:39:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 501.53056ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................XX.................] duration: 613.839883ms FAIL (2025-07-26 20:39:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 613.938842ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 340.651886ms FAIL (2025-07-26 20:39:38 UTC) orbiter warning: uplink: too many requests Duration: 340.690417ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................XX................] duration: 1.229020379s FAIL (2025-07-26 20:39:29 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.229059751s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................XX...............] duration: 5.678065255s FAIL (2025-07-26 20:39:16 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.678106654s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................XX................] duration: 617.321142ms FAIL (2025-07-26 20:39:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 617.379291ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 229.916796ms FAIL (2025-07-26 20:38:36 UTC) orbiter warning: uplink: too many requests Duration: 229.947435ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................XX.............] duration: 1.777260422s FAIL (2025-07-26 20:38:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.777289123s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................XX...............] duration: 469.330042ms FAIL (2025-07-26 20:38:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 469.352144ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................XX..............] duration: 569.245039ms FAIL (2025-07-26 20:38:09 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 569.276672ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................XX..............] duration: 551.121531ms FAIL (2025-07-26 20:38:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 551.14944ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 233.509241ms FAIL (2025-07-26 20:37:48 UTC) orbiter warning: uplink: too many requests Duration: 233.542879ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................XX.............] duration: 936.325293ms FAIL (2025-07-26 20:37:39 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 936.356772ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................XX............] duration: 1.26620585s FAIL (2025-07-26 20:37:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.266257962s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................XX............] duration: 1.068345487s FAIL (2025-07-26 20:37:21 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.068381691s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................XX.............] duration: 967.556815ms FAIL (2025-07-26 20:37:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 967.581691ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 247.550768ms FAIL (2025-07-26 20:36:55 UTC) orbiter warning: uplink: too many requests Duration: 247.583504ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................XX...........] duration: 613.643401ms FAIL (2025-07-26 20:36:20 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 614.476778ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................XX...........] duration: 515.741586ms FAIL (2025-07-26 20:36:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 515.778813ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................XX............] duration: 832.435551ms FAIL (2025-07-26 20:36:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 832.484163ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 250.296858ms FAIL (2025-07-26 20:35:42 UTC) orbiter warning: uplink: too many requests Duration: 250.348682ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................XX..........] duration: 528.227645ms FAIL (2025-07-26 20:35:39 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 528.263017ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................XX...........] duration: 708.392821ms FAIL (2025-07-26 20:35:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 708.527749ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................XX..........] duration: 439.010598ms FAIL (2025-07-26 20:35:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 439.046878ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................XX..........] duration: 552.552847ms FAIL (2025-07-26 20:35:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 552.584699ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................XX.........] duration: 1.034939428s FAIL (2025-07-26 20:34:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.034972323s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 285.802516ms FAIL (2025-07-26 20:34:52 UTC) orbiter warning: uplink: too many requests Duration: 285.831241ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................XX.........] duration: 1.461285702s FAIL (2025-07-26 20:34:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.461314713s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................XX.........] duration: 479.544044ms FAIL (2025-07-26 20:34:41 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 479.572521ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................XX........] duration: 397.270072ms FAIL (2025-07-26 20:34:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 397.298471ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 230.59404ms FAIL (2025-07-26 20:34:10 UTC) orbiter warning: uplink: too many requests Duration: 230.674962ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................XX.......] duration: 1.794340671s FAIL (2025-07-26 20:34:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.79437665s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................XX........] duration: 1.700947945s FAIL (2025-07-26 20:33:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.700975932s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................XX.......] duration: 1.344083018s FAIL (2025-07-26 20:33:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.344110345s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................XX........] duration: 1.076002837s FAIL (2025-07-26 20:33:29 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.076032049s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 494.065054ms FAIL (2025-07-26 20:33:22 UTC) orbiter warning: uplink: too many requests Duration: 494.167486ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................XX......] duration: 533.846131ms FAIL (2025-07-26 20:33:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 533.871611ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 340.803452ms FAIL (2025-07-26 20:33:07 UTC) orbiter warning: uplink: too many requests Duration: 340.832078ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................XX......] duration: 726.67056ms FAIL (2025-07-26 20:33:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 726.703884ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................XX.......] duration: 1.294809368s FAIL (2025-07-26 20:32:44 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.294839005s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 681.884291ms FAIL (2025-07-26 20:32:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 681.920705ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 214.76203ms FAIL (2025-07-26 20:32:20 UTC) orbiter warning: uplink: too many requests Duration: 214.796362ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 292.407455ms FAIL (2025-07-26 20:32:01 UTC) orbiter warning: uplink: too many requests Duration: 292.455879ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 500.270336ms FAIL (2025-07-26 20:31:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 500.323201ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................XX......] duration: 1.361894267s FAIL (2025-07-26 20:31:37 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.361916847s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 623.090159ms FAIL (2025-07-26 20:31:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 623.110989ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 1.105367657s FAIL (2025-07-26 20:31:15 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.105413065s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 1.507911397s FAIL (2025-07-26 20:31:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.50794937s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)