Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 372.817632ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:19:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 372.920351ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 276.115232ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:19:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 276.136986ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 484.734857ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:18:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 484.762247ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 249.662162ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:18:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 249.693587ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 449.504664ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:17:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 449.528243ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 240.610069ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:17:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 240.633914ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 413.759856ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:16:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 413.785263ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 244.452233ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:16:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 244.532661ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 456.535235ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:15:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 456.59725ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 257.146472ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:15:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 257.172472ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 393.589152ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:15:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 393.6159ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 412.043922ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:15:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 412.075781ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 253.520547ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:14:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 253.543892ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 508.11839ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:13:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 508.148547ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 269.678048ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:13:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 269.703345ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 281.690592ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:13:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 281.712433ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 430.899518ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:13:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 431.048398ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 415.672147ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:12:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 415.690888ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 259.319071ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:12:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 259.350958ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 382.463588ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:11:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 382.4996ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 409.735038ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:11:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 409.830301ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 762.846851ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:11:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 762.87031ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 420.525849ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:10:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 420.561411ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 635.015906ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:10:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 635.041507ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 501.369689ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:10:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 501.397401ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 789.609827ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:09:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 789.64535ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 526.433721ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:09:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 526.462866ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 541.128084ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:08:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 541.155974ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 330.96697ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:08:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 331.022184ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 386.446896ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:08:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 386.490109ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 493.542858ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:07:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 493.571123ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 581.16483ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:07:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 581.188404ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 249.420839ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:06:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 249.535828ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 571.595606ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:06:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 571.621889ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 280.218238ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:06:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 280.239751ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 411.082873ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:05:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 411.109034ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 288.834155ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:05:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 288.861231ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 411.766782ms trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:05:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 411.793163ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 263.703976ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:05:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 263.738991ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.022604723s trace: https://jaeger.dev.storj.io/trace/8dd3672e1c0360d FAIL (2025-12-07 05:05:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 7 05:04:32.465573464 (37h36m33.293281856s) [upload=39074]: Starting upload Dec 7 05:04:32.465665482 (37h36m33.293373874s) [upload=39074]: Done starting upload Dec 7 05:04:32.465885881 (37h36m33.293594264s) [upload=39074]: Uploading segments... Dec 7 05:04:32.466340720 (37h36m33.294049101s) [upload=39074]: Got next segment. Inline: false Dec 7 05:04:32.466410412 (37h36m33.294118793s) [upload=39074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 05:04:32.466415766 (37h36m33.294124158s) [upload=39074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 05:04:32.497829238 (37h36m33.32553763s) [seg_pos={0 0}, upload=39074]: Begin upload segment... Dec 7 05:04:32.497904702 (37h36m33.325613104s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498583053 (37h36m33.326291454s) [seg_pos={0 0}, upload=39074]: Done begin upload segment. Dec 7 05:04:32.498617015 (37h36m33.326325409s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.498834328 (37h36m33.326542722s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499155250 (37h36m33.326863641s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499447886 (37h36m33.327156283s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.499830386 (37h36m33.327538778s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500157568 (37h36m33.327865959s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.500501174 (37h36m33.328209568s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502269286 (37h36m33.329977697s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.502670266 (37h36m33.33037866s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503083792 (37h36m33.330792186s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503501451 (37h36m33.331209843s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.503630521 (37h36m33.331338912s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504253634 (37h36m33.331962027s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.504634490 (37h36m33.332342884s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.505370581 (37h36m33.333078974s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Uploading piece... Dec 7 05:04:32.533947086 (37h36m33.361655486s) [noise=true , address=77.90.31.21:4877 , piece_id=JNHYAZHGCHOA6QO3BEK7Q3JTGAIAV5WWZLLVDAUYS7K6WIGRMO2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.557901929 (37h36m33.385610327s) [noise=true , address=77.90.31.19:4841 , piece_id=AISK64YMSJALKXOIF5H5EQAXCOZFMTB3TY2OULQTWC3KQQHMCXOQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.562230060 (37h36m33.38993846s) [noise=true , address=77.90.31.21:4878 , piece_id=G6JTU5YC53UJR6EGLSZIJFUNEEEJS4NNHZGDOLROXOMFC5QGVJ6A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 7 05:04:32.574509906 (37h36m33.402218291s) [upload=39074]: Next returned nil segment Dec 7 05:04:32.574524147 (37h36m33.402232539s) [upload=39074]: Waiting for error group managing segments... Dec 7 05:04:32.581861320 (37h36m33.40956972s) [noise=true , address=77.90.31.12:4592 , piece_id=M6SEIHM36MABXV3BLQUPR4JB6X67DMIPJSIZP4XYWF7N4VUVAKDQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583620002 (37h36m33.411328393s) [noise=true , address=145.79.227.28:4725 , piece_id=FR7FR22PIZFDLSIWLSV5EGF5DTU4NVO7QG7KXNVZKVNWDRYGNCYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.583912914 (37h36m33.411621306s) [noise=true , address=145.79.227.34:4907 , piece_id=IEVOM26VQOOYLDXC66QLNQ6M4JRTZTS4BOMC5CORUDGUZ7JSFXYQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584512767 (37h36m33.412221159s) [noise=true , address=145.79.227.25:4619 , piece_id=6K6WWUEQ6WOILNP5T73RYLSTY4FULZ3TT7PEJ4VT7BBMUUVBUVJQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584686645 (37h36m33.412395028s) [noise=true , address=145.79.227.21:4475 , piece_id=PDC3SR3ZEIFQT3MOTQRHXC2VGUCQSEC7W2K6EFFIO2YD4F5KY42A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.584928862 (37h36m33.412637254s) [noise=true , address=145.79.227.39:4013 , piece_id=6UXVFUIBVL3334BOYO5YXW2W6QZ7FNYDOT4TF2UQMSYGI34WRSMA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.585351325 (37h36m33.413059717s) [noise=true , address=145.79.227.44:4195 , piece_id=UIUKV6WWZ775VLHLUP2NFJZMYJ6URW7BQVCJDL7YA4GIZPW2HV2A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.586282741 (37h36m33.413991132s) [noise=true , address=77.90.31.19:4842 , piece_id=3WJSP4XUJVMKPC27MC3PSD77HQJNNVTET72EAKQ3CPDL5YF3SPAQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589189290 (37h36m33.416897685s) [noise=true , address=77.90.31.13:4627 , piece_id=UM7CM6JZJCBOOXATUUPYPWR3N6OKWSEE5SFF6PUII2ULC3IHI3AQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.589302724 (37h36m33.417011122s) [noise=true , address=145.79.227.38:4979 , piece_id=M23FVLQYV7D4JOC7WATOJFH5VAQKUGO5CDSMBVGL7RGOVRDQZGZA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.590310070 (37h36m33.418018461s) [noise=true , address=145.79.227.20:4437 , piece_id=QETIT2Y7BD2KHCINRZF7ZUX6DWQ56CCDPE4IOMQ6UIBVGEY4PEBA, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:04:32.591375654 (37h36m33.419084045s) [noise=true , address=77.90.31.14:4661 , piece_id=L2I2RAJW3BX6H737IXTF5Y7KJVX2JS3UXJXND26WCZP4YQEIZK5A, seg_pos={0 0}, upload=39074]: Done uploading piece. err: <nil> Dec 7 05:05:02.434207605 (37h37m3.261916008s) [noise=true , address=77.90.31.16:4735 , piece_id=RLY2F2HI4VOEYHRFABZNFAJQIS4FUURFNPK5JMAJCX25MNWAXBQQ, seg_pos={0 0}, upload=39074]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 05:05:02.434263751 (37h37m3.261972146s) [seg_pos={0 0}, upload=39074]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 7 05:05:02.455114805 (37h37m3.282823204s) [upload=39074]: Done uploading segments... Dec 7 05:05:02.455161831 (37h37m3.282870233s) [upload=39074]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.022628196s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 252.070495ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:04:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 252.095265ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 261.815166ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:04:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 261.847415ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 246.112934ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:03:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 246.136238ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 243.156909ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:02:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 243.180876ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 249.403589ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:02:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 250.099023ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 256.862817ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:01:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 256.887022ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 354.762339ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:00:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 354.801009ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 331.411317ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 05:00:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 331.457545ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 237.666614ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 04:59:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 237.690336ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 262.633542ms trace: https://jaeger.dev.storj.io/trace/2f78c291ef823945 FAIL (2025-12-07 04:59:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 262.69069ms edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)